-- Logs begin at Tue 2022-10-04 07:59:27 UTC, end at Tue 2022-10-04 09:13:25 UTC. -- Oct 04 08:31:41.804380 dv-cinder-859447-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 04 08:31:41.813120 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Oct 4 08:31:41 2022] *** Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: nodename: dv-cinder-859447-2-iscsi Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: machine: x86_64 Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: clock source: unix Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: pcre jit disabled Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: detected number of CPU cores: 4 Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: current working directory: / Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: detected binary path: /usr/bin/uwsgi-core Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: your processes number limit is 47680 Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: your memory page size is 4096 bytes Oct 04 08:31:41.815008 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: detected max file descriptor number: 2048 Oct 04 08:31:41.816485 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: lock engine: pthread robust mutexes Oct 04 08:31:41.816485 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: thunder lock: enabled Oct 04 08:31:41.816485 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 04 08:31:41.816485 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 04 08:31:41.854985 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: Python main interpreter initialized at 0x5632691779c0 Oct 04 08:31:41.854985 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: python threads support enabled Oct 04 08:31:41.854985 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: your server socket listen backlog is limited to 100 connections Oct 04 08:31:41.854985 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: your mercy for graceful operations on workers is 90 seconds Oct 04 08:31:41.855336 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: mapped 403077 bytes (393 KB) for 2 cores Oct 04 08:31:41.855413 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: *** Operational MODE: preforking *** Oct 04 08:31:41.855675 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: *** uWSGI is running in multiple interpreter mode *** Oct 04 08:31:41.855675 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: spawned uWSGI master process (pid: 88190) Oct 04 08:31:41.855879 dv-cinder-859447-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 04 08:31:41.857218 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: spawned uWSGI worker 1 (pid: 88203, cores: 1) Oct 04 08:31:41.857218 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: spawned uWSGI worker 2 (pid: 88204, cores: 1) Oct 04 08:31:41.860051 dv-cinder-859447-2-iscsi devstack@keystone.service[88190]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 04 08:31:46.216161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:31:46.216161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: from cryptography.utils import int_from_bytes Oct 04 08:31:46.216943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:31:46.216943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: from cryptography.utils import int_from_bytes Oct 04 08:31:46.217360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:31:46.217360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: from cryptography.utils import int_from_bytes Oct 04 08:31:46.217465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:31:46.217465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: from cryptography.utils import int_from_bytes Oct 04 08:31:46.266775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 04 08:31:46.266946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 04 08:31:46.267036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 04 08:31:46.267124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] command line args: [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 04 08:31:46.267195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 04 08:31:46.267341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] command line args: [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 04 08:31:46.267425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 04 08:31:46.267557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ================================================================================ {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 04 08:31:46.267641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 04 08:31:46.267773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ================================================================================ {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 04 08:31:46.267853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] admin_token = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.268029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] admin_token = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.268644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] config_dir = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.268738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.268863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] config_dir = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.268997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] config_source = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.269129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.269239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] debug = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.269318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.269594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] config_source = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.269594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] debug = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.269785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.269893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] insecure_debug = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] insecure_debug = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] list_limit = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] log_config_append = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.270889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] list_limit = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] log_dir = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] log_config_append = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] log_file = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] log_dir = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] log_file = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.271940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.272970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.273893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.274059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.274148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.274303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.274459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] notification_format = cadf {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.274601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] notification_format = cadf {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.274685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.274842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] publish_errors = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] publish_errors = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.275977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] standard_threads = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] strict_password_check = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.276868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] standard_threads = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] strict_password_check = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] use_eventlog = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] use_journal = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] use_eventlog = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] use_json = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] use_journal = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] use_stderr = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] use_json = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] use_syslog = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.277974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] use_stderr = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.278048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] use_syslog = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.278192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] watch_log_file = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.278264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] watch_log_file = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:31:46.278331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.278538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.278617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.278774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.278865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.279990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.280947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281741 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.281973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.282961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.283124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.283201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.283355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.283436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.283605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.283687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.284571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.284661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.284878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.284975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.285882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.286990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.287944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.288937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.289862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.290949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.291953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.292878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.293965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294471 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.294932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.295900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.296901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.enabled = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.enabled = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.297998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.298961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.299919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.300947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.301883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.302966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.303927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] auth.external = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] auth.external = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] auth.mapped = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] auth.mapped = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.304936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] auth.password = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] auth.password = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] auth.token = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] auth.token = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.305909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] catalog.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] catalog.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.306895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] credential.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] credential.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.307861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] credential.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] credential.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.308973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.309983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.310156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.310235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.310451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.310550 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.310699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.310786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.310949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.311984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.312914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.313866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.314851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.315993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.316947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.317869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.318889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.319974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.320989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.321945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.password = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.322893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.password = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.323989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.324110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.324225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.324367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.324584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.324661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.324838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.324924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.325859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.326885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327653 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.327920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.328944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.329970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.330094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.330229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.330355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.331131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.331258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.331388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.331506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.331653 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.331788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.331788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.332899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.333026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.333175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.333304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.333431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.333560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.333685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.333806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] policy.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] policy.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.334958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] receipt.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] receipt.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.335905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.336983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337534 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] revoke.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] revoke.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.337959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.338036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] role.cache_time = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.338108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] role.cache_time = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.338206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] role.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.338286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] role.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] role.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] role.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] role.list_limit = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] role.list_limit = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.339996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.340954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.341892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.342251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.342338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.342511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.342591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.342761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.342840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.343988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.344928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.345985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.346134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.346211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.346992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.cache_time = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.cache_time = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.347955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.provider = fernet {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.provider = fernet {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.348924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] trust.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] trust.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.349960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.350982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351361 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.connection = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.351958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.connection = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.352897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.353836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.354975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.355945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.356099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.356176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.356256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88204) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 04 08:31:46.356328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:31:46.356486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=88203) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 04 08:31:46.385838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.386067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.386294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.386694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.386927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.387155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.395236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.395979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.423689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.423921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.424623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.424857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.478005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.479063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.526267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.526521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.527592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.527831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.682775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.683949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.684073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.706861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.707962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.708052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.708169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.708250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.708376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.708587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.719251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.719396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.732419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.732649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.732827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.732933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88203) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.733921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.734134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=88204) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:31:46.751593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.751701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.751803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:31:46.751885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:31:46.752148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.752243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.752417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:31:46.752495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:31:46.752703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.752881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.752981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.753220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.753301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.753482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.753561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.753804 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.763191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.763349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.763471 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.763626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.763788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.763921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.764051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.764203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.771432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.771606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.774287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.774500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.774585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.774785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.774867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.775988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.776171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.776249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.776443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.776524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.776714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.776793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.776978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.777056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.777245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.800344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.801711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.801801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.802052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.803198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.803486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.806674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.806956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.807238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.808115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.808400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.808685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.813466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.813754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.815019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.815367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.819349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.819640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.819923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.820206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.820482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.820756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.820988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.821069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.821279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.821358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.821565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.821851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.822128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.822419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.822693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.822964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.837023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.837300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.838746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.839029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.841056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.842862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.844312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.844609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.846047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.846926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.849312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.851104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.852366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.854205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.855500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.855799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.856083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.856363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.856640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.856918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.857398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.857699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.857988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.858277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.858586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.858883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.872698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.872984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.873250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.873511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.874749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.875041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.875314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.875579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.880429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.881954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.882569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.883486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.883780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.884113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.885645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.885925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.886500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.886786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.887068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.887351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.887748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.888608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.888895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.889180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.889464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.889862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.899344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.899628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.899910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.900192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.900571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.900859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.901138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.901652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.901986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.902274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.902582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.902969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.903261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.903555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.917070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.919444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.919960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.922321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.922836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.924433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.925236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.932095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.932095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.932095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.932334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.932334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.932334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.932334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.932918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.934364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.935148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.936592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.937221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.937615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.937902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.938182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.940497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.940899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.941190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.941471 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.946170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.946434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.946709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.949241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.949545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.949838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.953470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=88203) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.953799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.954130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.954431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.954706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.954994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.955267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.955541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.956604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.956604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.956880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.957173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=88204) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:31:46.957256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.957458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.957539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.957744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.957857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=88203) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.958114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.958606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.958884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.959163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.959433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.959710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.959997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.960275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.960542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.960812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=88204) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:31:46.993080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5632691779c0 pid: 88203 (default app) Oct 04 08:31:46.995124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5632691779c0 pid: 88204 (default app) Oct 04 08:31:46.996441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a11a8d1-aa61-4ad3-8da8-690997e9eab6 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:46.996725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a11a8d1-aa61-4ad3-8da8-690997e9eab6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:46.996997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a11a8d1-aa61-4ad3-8da8-690997e9eab6 None None] PATH_INFO: `/v3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:47.001204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Tue Oct 4 08:31:46 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:31:55.112148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf13c4-9c9b-4346-843a-a5c7cec775d9 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:55.113199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf13c4-9c9b-4346-843a-a5c7cec775d9 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:55.113199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf13c4-9c9b-4346-843a-a5c7cec775d9 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:55.117801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:31:55 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:31:55.122955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:55.123476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:55.123476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:55.193864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: INFO dbcounter [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] Registered counter for database keystone Oct 04 08:31:55.214345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG oslo_db.sqlalchemy.engines [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88203) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 04 08:31:55.220650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writer thread running {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 04 08:31:55.405356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG passlib.handlers.bcrypt [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=88203) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 04 08:31:55.424392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG passlib.handlers.bcrypt [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=88203) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 04 08:31:55.465078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:31:55.466716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:55.598788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:31:55 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 477 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 04 08:31:56.075336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbf13c4-9c9b-4346-843a-a5c7cec775d9 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:56.146182 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: INFO dbcounter [None req-2cbf13c4-9c9b-4346-843a-a5c7cec775d9 None None] Registered counter for database keystone Oct 04 08:31:56.165160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG oslo_db.sqlalchemy.engines [None req-2cbf13c4-9c9b-4346-843a-a5c7cec775d9 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88204) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 04 08:31:56.171335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writer thread running {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 04 08:31:56.292492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbf13c4-9c9b-4346-843a-a5c7cec775d9 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:56.296806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192eb60f-0ba0-4122-a0c6-e5717974ac41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:56.297433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192eb60f-0ba0-4122-a0c6-e5717974ac41 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:56.297710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192eb60f-0ba0-4122-a0c6-e5717974ac41 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:56.297978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192eb60f-0ba0-4122-a0c6-e5717974ac41 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:56.307127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192eb60f-0ba0-4122-a0c6-e5717974ac41 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:56.359676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING keystone.common.rbac_enforcer.enforcer [None req-192eb60f-0ba0-4122-a0c6-e5717974ac41 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 04 08:31:56.359960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192eb60f-0ba0-4122-a0c6-e5717974ac41 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:56.370219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Tue Oct 4 08:31:56 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 08:31:56.376451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:56.393611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f1a9f0-0245-406f-83fd-a9e001e85f7c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:56.398107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884421f8-c44b-4db9-bb76-a859b43cad89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:56.399245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884421f8-c44b-4db9-bb76-a859b43cad89 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:56.399528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884421f8-c44b-4db9-bb76-a859b43cad89 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:56.399809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884421f8-c44b-4db9-bb76-a859b43cad89 admin admin] PATH_INFO: `/v3/projects/24e9902302e04e8e8d4caf3bfee360d5` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:56.401385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884421f8-c44b-4db9-bb76-a859b43cad89 admin admin] RBAC: Authorizing `identity:get_project(project_id=24e9902302e04e8e8d4caf3bfee360d5)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:56.454703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING keystone.common.rbac_enforcer.enforcer [None req-884421f8-c44b-4db9-bb76-a859b43cad89 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 04 08:31:56.455053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884421f8-c44b-4db9-bb76-a859b43cad89 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:56.459943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Tue Oct 4 08:31:56 2022] GET /identity/v3/projects/24e9902302e04e8e8d4caf3bfee360d5 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:31:58.583483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503a90b0-4e79-4502-9bad-11b147b3ed9b None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:58.584004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503a90b0-4e79-4502-9bad-11b147b3ed9b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:58.584004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503a90b0-4e79-4502-9bad-11b147b3ed9b None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:58.588058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:31:58 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:31:58.593778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa5d69d-a5fe-48a0-800a-20dd26fc8afd None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:58.594151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa5d69d-a5fe-48a0-800a-20dd26fc8afd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:58.594327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa5d69d-a5fe-48a0-800a-20dd26fc8afd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:58.614637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-daa5d69d-a5fe-48a0-800a-20dd26fc8afd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:31:58.616184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-daa5d69d-a5fe-48a0-800a-20dd26fc8afd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:58.624282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:31:58 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 04 08:31:58.903333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503a90b0-4e79-4502-9bad-11b147b3ed9b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:58.915404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503a90b0-4e79-4502-9bad-11b147b3ed9b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:58.919597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2bc981-ba0d-438c-917b-d661d163170c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:58.920199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2bc981-ba0d-438c-917b-d661d163170c admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:58.920476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2bc981-ba0d-438c-917b-d661d163170c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:58.920745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2bc981-ba0d-438c-917b-d661d163170c admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:58.928909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2bc981-ba0d-438c-917b-d661d163170c admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:58.930693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2bc981-ba0d-438c-917b-d661d163170c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:58.939224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Tue Oct 4 08:31:58 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:31:58.945011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa5d69d-a5fe-48a0-800a-20dd26fc8afd None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:58.953979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa5d69d-a5fe-48a0-800a-20dd26fc8afd None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:58.958317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a135af-d387-424c-989c-d8881f6c5a8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:58.958982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a135af-d387-424c-989c-d8881f6c5a8c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:58.959263 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a135af-d387-424c-989c-d8881f6c5a8c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:58.959538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a135af-d387-424c-989c-d8881f6c5a8c admin admin] PATH_INFO: `/v3/users/e79e89b7261f4751ae9cd79132ff7ae2` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:58.960841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a135af-d387-424c-989c-d8881f6c5a8c admin admin] RBAC: Authorizing `identity:get_user(user_id=e79e89b7261f4751ae9cd79132ff7ae2)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:58.962680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a135af-d387-424c-989c-d8881f6c5a8c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:58.964183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Tue Oct 4 08:31:58 2022] GET /identity/v3/users/e79e89b7261f4751ae9cd79132ff7ae2 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:01.207068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baaafc6-4aad-4d77-b5d0-37b81c5ce57f None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:01.207583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baaafc6-4aad-4d77-b5d0-37b81c5ce57f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:01.207655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baaafc6-4aad-4d77-b5d0-37b81c5ce57f None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:01.208933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:01.214472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e01a05-4446-4304-9638-0755814d38b6 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:01.214890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e01a05-4446-4304-9638-0755814d38b6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:01.215097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e01a05-4446-4304-9638-0755814d38b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:01.225849 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e96b27-9f44-41ad-9651-e97d6e804c6b None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:01.226127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e96b27-9f44-41ad-9651-e97d6e804c6b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:01.226397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e96b27-9f44-41ad-9651-e97d6e804c6b None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:01.227701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:01.232759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:01.233033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:01.233125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-64e01a05-4446-4304-9638-0755814d38b6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:01.233299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:01.234217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-64e01a05-4446-4304-9638-0755814d38b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:01.275294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG passlib.handlers.bcrypt [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=88204) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 04 08:32:01.275674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG passlib.handlers.bcrypt [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=88204) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 04 08:32:01.314032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:01.325504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:01.327120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:01.429276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:01.579941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e01a05-4446-4304-9638-0755814d38b6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:01.603471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:01.682520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e01a05-4446-4304-9638-0755814d38b6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:01.696045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:01.717526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:01.718152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:01.718456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:01.718729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:01.730840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:01.740194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:01.741241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:01.742201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:01.744402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:01.751196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:01.801143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c16d49-2599-48f6-a79d-7d746c7b3c41 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:01.803612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:01.840700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:01.841971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:01.842857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:01.843188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:01.843444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:01.858111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12970451-69ee-4e82-8e2a-8886e58ad80f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:01.894264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:01.894989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:01.895662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:01.895952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:01.896613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:01.896916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:01.898307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:01.909681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:01.910384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:01.911993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:01.916057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:02.010217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 8/16] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:32:02.013663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f421a-104a-4786-b3f7-2a5de83190ef None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:02.015682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:02.018281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:02.048208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:02.048836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:02.049112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:02.049382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] PATH_INFO: `/v3/users/e79e89b7261f4751ae9cd79132ff7ae2` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:02.051285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] RBAC: Authorizing `identity:get_user(user_id=e79e89b7261f4751ae9cd79132ff7ae2)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:02.052742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:02.054321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:01 2022] GET /identity/v3/users/e79e89b7261f4751ae9cd79132ff7ae2 => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:02.059912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:02.075585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:02.121090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a996be-63dc-4018-bbc5-c74cd268d1b9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:02.154886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:02.155745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:02.156028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:02.156297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:02.158823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12bb08-df83-4a3a-9bdd-266e801cabcd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:02.174750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:02.176826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:02.193163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:02.193795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:02.194076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:02.194347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:02.195498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:32:02.196013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:02.196936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:32:02.197789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:32:02.197789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:32:02.197789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:02.197941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:02.198008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 9/18] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:02.198547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 10/19] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:02.203641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:02.204163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:02.260313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:02.287607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:02.304658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7beec0-324a-477c-ae7c-c08cdbe739e7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:02.305850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342aacd0-6e28-4f2d-b7aa-7ddd6b3fd95c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:02.340199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ddca70-6e0f-4742-b8f5-1b3289694560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:02.340473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32799a45-17e7-4ea1-814d-c3f1038ccb70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:02.340877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ddca70-6e0f-4742-b8f5-1b3289694560 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:02.341061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32799a45-17e7-4ea1-814d-c3f1038ccb70 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:02.341165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ddca70-6e0f-4742-b8f5-1b3289694560 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:02.341342 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32799a45-17e7-4ea1-814d-c3f1038ccb70 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:02.341426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ddca70-6e0f-4742-b8f5-1b3289694560 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:02.341613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32799a45-17e7-4ea1-814d-c3f1038ccb70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:02.342050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ddca70-6e0f-4742-b8f5-1b3289694560 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:02.342332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32799a45-17e7-4ea1-814d-c3f1038ccb70 None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.domain.id=default, user.id=e79e89b7261f4751ae9cd79132ff7ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:02.342455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ddca70-6e0f-4742-b8f5-1b3289694560 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:02.342670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32799a45-17e7-4ea1-814d-c3f1038ccb70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:02.343753 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ddca70-6e0f-4742-b8f5-1b3289694560 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:02.344105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32799a45-17e7-4ea1-814d-c3f1038ccb70 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:02.351224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 11/20] 10.222.0.27 () {56 vars in 1322 bytes} [Tue Oct 4 08:32:02 2022] GET /identity/v3/role_assignments?user.id=e79e89b7261f4751ae9cd79132ff7ae2&scope.domain.id=default&role.id=1922dc727b87489999cc8916eacc229e => generated 233 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:02.362308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 10/21] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:32:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:04.992768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ed42b-af1a-46b3-a60e-2665c3753cc7 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:04.993758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ed42b-af1a-46b3-a60e-2665c3753cc7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:04.994229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ed42b-af1a-46b3-a60e-2665c3753cc7 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:04.995782 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 12/22] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:05.001285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbddfc41-1b8c-4197-bacc-d710391303de None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.002114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbddfc41-1b8c-4197-bacc-d710391303de None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.002633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbddfc41-1b8c-4197-bacc-d710391303de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.026481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-bbddfc41-1b8c-4197-bacc-d710391303de None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:05.028036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bbddfc41-1b8c-4197-bacc-d710391303de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:05.121249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 11/23] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:05.236994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe7d9a-7ac6-4458-8730-4ba432008e80 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.238285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe7d9a-7ac6-4458-8730-4ba432008e80 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.238998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe7d9a-7ac6-4458-8730-4ba432008e80 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.240538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 13/24] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:05.246242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e31b21-5833-4f8f-9d2c-38c79c1b6bf1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.246788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e31b21-5833-4f8f-9d2c-38c79c1b6bf1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.247256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e31b21-5833-4f8f-9d2c-38c79c1b6bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.271471 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-02e31b21-5833-4f8f-9d2c-38c79c1b6bf1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:05.272670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-02e31b21-5833-4f8f-9d2c-38c79c1b6bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:05.312672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f1d01-b8f1-4bdc-ac62-6ce2b80e1959 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.313306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f1d01-b8f1-4bdc-ac62-6ce2b80e1959 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.313785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f1d01-b8f1-4bdc-ac62-6ce2b80e1959 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.315442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 14/25] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:05.321553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd098d52-509a-4014-b841-cc7af65debe4 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.322167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd098d52-509a-4014-b841-cc7af65debe4 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.322685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd098d52-509a-4014-b841-cc7af65debe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.323848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:05.352736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-bd098d52-509a-4014-b841-cc7af65debe4 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:05.354305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-bd098d52-509a-4014-b841-cc7af65debe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:05.364266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 12/26] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:05.464043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 15/27] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:05.532460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ce28ec-2bf4-424e-bd82-c0def536feeb None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.533072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ce28ec-2bf4-424e-bd82-c0def536feeb None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.533558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ce28ec-2bf4-424e-bd82-c0def536feeb None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.536070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd098d52-509a-4014-b841-cc7af65debe4 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:05.537693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 13/28] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:05 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:05.555099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cdbc8c-ba0a-4e0a-a9b5-394c5f73f453 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.555683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cdbc8c-ba0a-4e0a-a9b5-394c5f73f453 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.556173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cdbc8c-ba0a-4e0a-a9b5-394c5f73f453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.584462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e8cdbc8c-ba0a-4e0a-a9b5-394c5f73f453 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:05.586381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e8cdbc8c-ba0a-4e0a-a9b5-394c5f73f453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:05.657986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:05.709211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd098d52-509a-4014-b841-cc7af65debe4 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:05.744168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 14/29] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:05.771578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:05.772593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:05.773107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:05.773617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:05.794954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:05.796775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:05.801155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cdbc8c-ba0a-4e0a-a9b5-394c5f73f453 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:32:05.846390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:05.848012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:32:05.848898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:32:05.848898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:32:05.848898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:05.850942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 16/30] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:05.856447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:06.016739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cdbc8c-ba0a-4e0a-a9b5-394c5f73f453 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:06.028060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a12d8e-5f95-4cd8-baf9-35e725fe6552 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:06.028931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:06.062706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:06.073984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:06.074932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.075541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.076019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.076905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:06.077391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:06.078998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:06.092289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:06.092912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.093189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.093461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.094149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:06.095564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:06.100682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 17/31] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:32:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:06.101748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:06.213509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531c2e5b-557e-4c83-82d8-11506b8cde8d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:06.252207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:06.253773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.254299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.254769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.255721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:06.257284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:06.302287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 15/32] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Oct 4 08:32:05 2022] POST /identity/v3/roles => generated 225 bytes in 501 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:32:06.306583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:06.442451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e21ba06-e6a7-4574-ab08-5319a3b28670 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:06.489652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:06.490348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.490678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.490955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.503523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:06.510662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:06.511482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:06.512206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 16/33] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:06.512804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:06.519663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 18/34] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Oct 4 08:32:06 2022] POST /identity/v3/roles => generated 219 bytes in 418 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:06.526525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:06.578777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:06.625618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:06.687651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4d6043-ce2a-42e0-9636-7c3df1a25a07 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:06.720302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf25190-31bc-4349-9aed-26d3484cef07 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:06.749392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8993d95d-8034-4aba-bf4f-3d9f7d922162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:06.750031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993d95d-8034-4aba-bf4f-3d9f7d922162 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.750316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993d95d-8034-4aba-bf4f-3d9f7d922162 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.750742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993d95d-8034-4aba-bf4f-3d9f7d922162 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.751824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8993d95d-8034-4aba-bf4f-3d9f7d922162 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:06.753988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8993d95d-8034-4aba-bf4f-3d9f7d922162 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:06.770224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcc0021-e83c-4262-b427-35ef236a611e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:06.770895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcc0021-e83c-4262-b427-35ef236a611e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.771699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcc0021-e83c-4262-b427-35ef236a611e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.771699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcc0021-e83c-4262-b427-35ef236a611e None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.772292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcc0021-e83c-4262-b427-35ef236a611e None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:06.772595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcc0021-e83c-4262-b427-35ef236a611e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:06.774113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcc0021-e83c-4262-b427-35ef236a611e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:06.783967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 19/35] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:06.788306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aead95b-90a7-4256-bbcc-f497110f7882 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.788306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aead95b-90a7-4256-bbcc-f497110f7882 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.788306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aead95b-90a7-4256-bbcc-f497110f7882 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.788306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 20/36] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:06.788306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373942aa-ceaa-4a14-a64a-44e581641e1d None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.788629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373942aa-ceaa-4a14-a64a-44e581641e1d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.788823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373942aa-ceaa-4a14-a64a-44e581641e1d None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.789990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 21/37] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:06.790713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373942aa-ceaa-4a14-a64a-44e581641e1d None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:06.898312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373942aa-ceaa-4a14-a64a-44e581641e1d None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:06.907159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:06.920909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 17/38] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:32:06 2022] POST /identity/v3/projects => generated 301 bytes in 409 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:06.922387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ffca03-e29c-4555-835a-878e2f3da2ca None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.922701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ffca03-e29c-4555-835a-878e2f3da2ca None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.922976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ffca03-e29c-4555-835a-878e2f3da2ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.936431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45802ded-4eb8-4a90-a25a-b65589b32894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:06.937183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45802ded-4eb8-4a90-a25a-b65589b32894 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.937742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45802ded-4eb8-4a90-a25a-b65589b32894 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.937742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45802ded-4eb8-4a90-a25a-b65589b32894 None admin] PATH_INFO: `/v3/users/e79e89b7261f4751ae9cd79132ff7ae2` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.939218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45802ded-4eb8-4a90-a25a-b65589b32894 None admin] RBAC: Authorizing `identity:get_user(user_id=e79e89b7261f4751ae9cd79132ff7ae2)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:06.940735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45802ded-4eb8-4a90-a25a-b65589b32894 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:06.942464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 22/39] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:06 2022] GET /identity/v3/users/e79e89b7261f4751ae9cd79132ff7ae2 => generated 247 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:06.943822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455c2416-e4e4-4ec6-bbc2-4f055725d22e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:06.944110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455c2416-e4e4-4ec6-bbc2-4f055725d22e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:06.944388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455c2416-e4e4-4ec6-bbc2-4f055725d22e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:06.956265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-82ffca03-e29c-4555-835a-878e2f3da2ca None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:06.959033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-82ffca03-e29c-4555-835a-878e2f3da2ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:06.964164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-455c2416-e4e4-4ec6-bbc2-4f055725d22e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:06.965762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-455c2416-e4e4-4ec6-bbc2-4f055725d22e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:07.040297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 23/40] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:07.041440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455c2416-e4e4-4ec6-bbc2-4f055725d22e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:07.042267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 18/41] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:07.164985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455c2416-e4e4-4ec6-bbc2-4f055725d22e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:07.173163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:07.204707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:07.205283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:07.205566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:07.205844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:07.207731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:07.209238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:07.215268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 24/42] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:07.217061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ffca03-e29c-4555-835a-878e2f3da2ca None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:07.232608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:07.319501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:07.325077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ffca03-e29c-4555-835a-878e2f3da2ca None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:07.361654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:07.362377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:07.362889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:07.363251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] PATH_INFO: `/v3/domains/default/users/e79e89b7261f4751ae9cd79132ff7ae2/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:07.365868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=e79e89b7261f4751ae9cd79132ff7ae2, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:07.367378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:07.453154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 19/43] 10.222.0.27 () {58 vars in 1361 bytes} [Tue Oct 4 08:32:07 2022] PUT /identity/v3/domains/default/users/e79e89b7261f4751ae9cd79132ff7ae2/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:07.453983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:07.539935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50884f52-5b14-402a-98fd-c61e95aa377f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:07.548111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:07.580058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:07.580713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:07.580993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:07.581896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:07.583189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:07.584772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:07.586533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 25/44] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:07.589790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec93aa-02b2-46ba-9f37-d6298ae15947 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:07.592168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:07.631217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9c6ad3-b7c9-4c02-8a35-d895c42e031f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:07.631416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c6ad3-b7c9-4c02-8a35-d895c42e031f None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:07.631644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c6ad3-b7c9-4c02-8a35-d895c42e031f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:07.631829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c6ad3-b7c9-4c02-8a35-d895c42e031f None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:07.632782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9c6ad3-b7c9-4c02-8a35-d895c42e031f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:07.633878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9c6ad3-b7c9-4c02-8a35-d895c42e031f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:07.646105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:07.701726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c7fee3-96ee-4fb6-8da1-c29d8c330ca0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:07.725487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 20/45] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Oct 4 08:32:07 2022] POST /identity/v3/roles => generated 223 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:07.736805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6043d7-20cb-4b87-88cb-8dbeb144a741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:07.737745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6043d7-20cb-4b87-88cb-8dbeb144a741 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:07.738025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6043d7-20cb-4b87-88cb-8dbeb144a741 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:07.738348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6043d7-20cb-4b87-88cb-8dbeb144a741 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:07.739075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6043d7-20cb-4b87-88cb-8dbeb144a741 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:07.740575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6043d7-20cb-4b87-88cb-8dbeb144a741 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:07.827376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 26/46] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:32:07 2022] POST /identity/v3/projects => generated 312 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:32:09.637413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbe82ea-cdc0-4c7c-b932-8cc1db3d36f2 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:09.637922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbe82ea-cdc0-4c7c-b932-8cc1db3d36f2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:09.638030 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbe82ea-cdc0-4c7c-b932-8cc1db3d36f2 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:09.639323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 21/47] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:09.645424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082c7e-829a-4630-96d8-5f650ac4a063 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:09.645902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082c7e-829a-4630-96d8-5f650ac4a063 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:09.645902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082c7e-829a-4630-96d8-5f650ac4a063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:09.664783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-75082c7e-829a-4630-96d8-5f650ac4a063 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:09.665998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-75082c7e-829a-4630-96d8-5f650ac4a063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:09.731501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 27/48] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:10.006172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbe82ea-cdc0-4c7c-b932-8cc1db3d36f2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:10.044469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f914e17f-d5aa-4905-bc2f-04aba228ef67 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.044751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f914e17f-d5aa-4905-bc2f-04aba228ef67 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.045025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f914e17f-d5aa-4905-bc2f-04aba228ef67 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.046357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 28/49] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:10.051912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb8e472-1657-4a36-8b8e-002b1963e59b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.052258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb8e472-1657-4a36-8b8e-002b1963e59b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.052596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb8e472-1657-4a36-8b8e-002b1963e59b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.070573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-2cb8e472-1657-4a36-8b8e-002b1963e59b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:10.071748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2cb8e472-1657-4a36-8b8e-002b1963e59b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:10.093320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:10.110237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbe82ea-cdc0-4c7c-b932-8cc1db3d36f2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:10.137062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:10.143714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 29/50] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:10.144701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:10.145339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.145616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.145887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.156071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:10.161721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:10.163339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:10.164829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:10.167706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 22/51] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:10.173128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb8e472-1657-4a36-8b8e-002b1963e59b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:10.282519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb8e472-1657-4a36-8b8e-002b1963e59b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:10.316356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:10.316973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.317258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.317536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.318203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:10.318531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:10.319919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:10.327698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 30/52] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:10.334467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:10.416253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:10.434471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:10.435758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3a3171-88d1-4b60-bfd3-d8944bb7ac9a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:10.469337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cebef-6583-497a-8843-404107c21819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:10.469977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cebef-6583-497a-8843-404107c21819 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.470255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cebef-6583-497a-8843-404107c21819 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.470565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cebef-6583-497a-8843-404107c21819 None admin] PATH_INFO: `/v3/users/e79e89b7261f4751ae9cd79132ff7ae2` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.472124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22cebef-6583-497a-8843-404107c21819 None admin] RBAC: Authorizing `identity:get_user(user_id=e79e89b7261f4751ae9cd79132ff7ae2)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:10.473754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22cebef-6583-497a-8843-404107c21819 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:10.475380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 23/53] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:10 2022] GET /identity/v3/users/e79e89b7261f4751ae9cd79132ff7ae2 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:10.480619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cebef-6583-497a-8843-404107c21819 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:10.517456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d13f6f0-e536-42c2-a81e-bb38b2f42d97 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:10.537106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:10.551785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:10.552410 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.552693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.552968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.554079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:10.555590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:10.557180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 31/54] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:10.562636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:10.580740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cebef-6583-497a-8843-404107c21819 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:10.615490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:10.616131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.616411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.616741 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.618121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:10.619625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:10.621183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 24/55] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:10.626914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:10.647835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:10.663577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6caa03-75a0-4f9f-9c30-f3d41383a0a5 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:10.698021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ba0224-ff0e-4911-ab5e-6bb6d50dc6d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:10.698655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba0224-ff0e-4911-ab5e-6bb6d50dc6d3 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.698938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba0224-ff0e-4911-ab5e-6bb6d50dc6d3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.699209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ba0224-ff0e-4911-ab5e-6bb6d50dc6d3 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.699851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ba0224-ff0e-4911-ab5e-6bb6d50dc6d3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:10.701286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ba0224-ff0e-4911-ab5e-6bb6d50dc6d3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:10.820777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 32/56] 10.222.0.27 () {60 vars in 1140 bytes} [Tue Oct 4 08:32:10 2022] POST /identity/v3/projects => generated 298 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:32:10.837849 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec062622-696e-42b9-99d4-71aee18949b0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:10.865722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:10.876916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f368f9-3ef7-4db1-89a0-c46e74eb9a8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:10.877554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f368f9-3ef7-4db1-89a0-c46e74eb9a8b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:10.877834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f368f9-3ef7-4db1-89a0-c46e74eb9a8b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:10.878147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f368f9-3ef7-4db1-89a0-c46e74eb9a8b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:10.879475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f368f9-3ef7-4db1-89a0-c46e74eb9a8b None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.domain.id=default, user.id=e79e89b7261f4751ae9cd79132ff7ae2) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:10.886167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f368f9-3ef7-4db1-89a0-c46e74eb9a8b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:10.886167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f368f9-3ef7-4db1-89a0-c46e74eb9a8b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:10.890226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 25/57] 10.222.0.27 () {56 vars in 1322 bytes} [Tue Oct 4 08:32:10 2022] GET /identity/v3/role_assignments?user.id=e79e89b7261f4751ae9cd79132ff7ae2&scope.domain.id=default&role.id=1922dc727b87489999cc8916eacc229e => generated 530 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:12.992997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8015e702-6e17-436e-bafe-af528941d693 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:12.993537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8015e702-6e17-436e-bafe-af528941d693 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:12.993537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8015e702-6e17-436e-bafe-af528941d693 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:12.994910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 33/58] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:13.000065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d6584f-6e38-412c-b5e6-d4b06284361f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:13.000473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d6584f-6e38-412c-b5e6-d4b06284361f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:13.000617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d6584f-6e38-412c-b5e6-d4b06284361f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:13.018245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-02d6584f-6e38-412c-b5e6-d4b06284361f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:13.019526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-02d6584f-6e38-412c-b5e6-d4b06284361f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:13.083334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 26/59] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:13.353206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8015e702-6e17-436e-bafe-af528941d693 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:13.452130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8015e702-6e17-436e-bafe-af528941d693 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:13.484151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc82724-461d-433a-80fb-38a0121a3740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:13.484768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc82724-461d-433a-80fb-38a0121a3740 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:13.485049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc82724-461d-433a-80fb-38a0121a3740 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:13.485322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc82724-461d-433a-80fb-38a0121a3740 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:13.486432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc82724-461d-433a-80fb-38a0121a3740 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:13.488556 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc82724-461d-433a-80fb-38a0121a3740 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:13.490086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 34/60] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:13.495862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d6584f-6e38-412c-b5e6-d4b06284361f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:13.591166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d6584f-6e38-412c-b5e6-d4b06284361f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:13.608464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:13.625368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d95753-38af-401c-b2ab-5357fe90b6c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:13.625978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d95753-38af-401c-b2ab-5357fe90b6c2 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:13.626262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d95753-38af-401c-b2ab-5357fe90b6c2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:13.626566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d95753-38af-401c-b2ab-5357fe90b6c2 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:13.627228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d95753-38af-401c-b2ab-5357fe90b6c2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:13.628691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d95753-38af-401c-b2ab-5357fe90b6c2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:13.738620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 27/61] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:32:13 2022] POST /identity/v3/users => generated 275 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:32:17.104509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0696a50-3b3c-4c47-87f2-372a14d77300 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.105539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0696a50-3b3c-4c47-87f2-372a14d77300 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.106027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0696a50-3b3c-4c47-87f2-372a14d77300 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.107670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 35/62] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:17.114522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb8eff-fc57-4d5c-afb0-154274b924f4 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.115351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb8eff-fc57-4d5c-afb0-154274b924f4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.115827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb8eff-fc57-4d5c-afb0-154274b924f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.144050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-cdcb8eff-fc57-4d5c-afb0-154274b924f4 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:17.146204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cdcb8eff-fc57-4d5c-afb0-154274b924f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:17.172657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2424e125-ea1a-4b79-a5a0-a61072c11da2 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.173257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2424e125-ea1a-4b79-a5a0-a61072c11da2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.173728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2424e125-ea1a-4b79-a5a0-a61072c11da2 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.175977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 36/63] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:17.181119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c9fec-5636-4057-98b1-7db376e407f8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.181635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c9fec-5636-4057-98b1-7db376e407f8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.182131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c9fec-5636-4057-98b1-7db376e407f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.194146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:17.211980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ba2c9fec-5636-4057-98b1-7db376e407f8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:17.219833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ba2c9fec-5636-4057-98b1-7db376e407f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:17.255139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 28/64] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:17.343952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 37/65] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:17.560847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb8eff-fc57-4d5c-afb0-154274b924f4 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:17.576553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91e4db7-d22b-48db-af5a-a7bc43fc3160 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.577273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91e4db7-d22b-48db-af5a-a7bc43fc3160 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.577844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91e4db7-d22b-48db-af5a-a7bc43fc3160 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.579715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 38/66] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:17.585222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b854f5b-6293-43eb-af25-6f0f3988c430 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.585835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b854f5b-6293-43eb-af25-6f0f3988c430 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.586307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b854f5b-6293-43eb-af25-6f0f3988c430 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.615520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9b854f5b-6293-43eb-af25-6f0f3988c430 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:17.617168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9b854f5b-6293-43eb-af25-6f0f3988c430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:17.675510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb8eff-fc57-4d5c-afb0-154274b924f4 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:17.679052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:17.699046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 39/67] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:17.700416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51534fe5-9911-4637-83c6-4f9de7dae884 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.700710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51534fe5-9911-4637-83c6-4f9de7dae884 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.700991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51534fe5-9911-4637-83c6-4f9de7dae884 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.702234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 40/68] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:17.707616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51534fe5-9911-4637-83c6-4f9de7dae884 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:17.720344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57ef43b-4f8b-465f-bde5-0da50df7ac67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:17.721641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57ef43b-4f8b-465f-bde5-0da50df7ac67 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.723479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57ef43b-4f8b-465f-bde5-0da50df7ac67 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.724241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57ef43b-4f8b-465f-bde5-0da50df7ac67 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.737048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57ef43b-4f8b-465f-bde5-0da50df7ac67 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-a57ef43b-4f8b-465f-bde5-0da50df7ac67 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:17.743815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:17.744759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:17.745501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:17.748037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 29/69] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:17.752998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09291b6-3a8c-4bd2-ada9-501af35bcba1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.752998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09291b6-3a8c-4bd2-ada9-501af35bcba1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.753982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09291b6-3a8c-4bd2-ada9-501af35bcba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.775974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b09291b6-3a8c-4bd2-ada9-501af35bcba1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:17.777203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b09291b6-3a8c-4bd2-ada9-501af35bcba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:17.793386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:17.852310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51534fe5-9911-4637-83c6-4f9de7dae884 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:17.858101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 30/70] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:17.859169 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09291b6-3a8c-4bd2-ada9-501af35bcba1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:17.892437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304504b5-33e2-40a3-9a15-4d1238cad112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:17.893243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304504b5-33e2-40a3-9a15-4d1238cad112 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.894526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304504b5-33e2-40a3-9a15-4d1238cad112 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.895525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304504b5-33e2-40a3-9a15-4d1238cad112 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.908831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304504b5-33e2-40a3-9a15-4d1238cad112 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-304504b5-33e2-40a3-9a15-4d1238cad112 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:17.914713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:17.915591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:17.916318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:17.918687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 41/71] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:17.921816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b6f5e-382d-4d7b-a269-060351d2c0e5 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:17.922636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b6f5e-382d-4d7b-a269-060351d2c0e5 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:17.923374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b6f5e-382d-4d7b-a269-060351d2c0e5 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:17.926844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 42/72] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:17.928578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20b6f5e-382d-4d7b-a269-060351d2c0e5 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:17.978601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:17.985227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09291b6-3a8c-4bd2-ada9-501af35bcba1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.022979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861e47cd-9566-4025-a1bc-bf9847dd65dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.023929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861e47cd-9566-4025-a1bc-bf9847dd65dc None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.024204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861e47cd-9566-4025-a1bc-bf9847dd65dc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.024618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861e47cd-9566-4025-a1bc-bf9847dd65dc None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.025421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861e47cd-9566-4025-a1bc-bf9847dd65dc None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:18.025865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861e47cd-9566-4025-a1bc-bf9847dd65dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.031143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861e47cd-9566-4025-a1bc-bf9847dd65dc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.038336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20b6f5e-382d-4d7b-a269-060351d2c0e5 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.045457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 31/73] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:18.047259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11ead39-a981-4147-8ee8-3a4709250a7f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.047571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11ead39-a981-4147-8ee8-3a4709250a7f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.047846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11ead39-a981-4147-8ee8-3a4709250a7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.067027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e11ead39-a981-4147-8ee8-3a4709250a7f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:18.068327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e11ead39-a981-4147-8ee8-3a4709250a7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:18.068818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.081226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.081842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.082124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.082394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.083084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:18.083387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.084770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.092855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 43/74] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:18.093490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.140025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 32/75] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:18.140938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11ead39-a981-4147-8ee8-3a4709250a7f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.214314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef32a542-341e-4402-8a83-3e0a3cd278fa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.242293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.249778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.250424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.250770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.251066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.257581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11ead39-a981-4147-8ee8-3a4709250a7f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.263210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:18.269176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:18.270135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 44/76] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:18.275374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.300780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.301427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.301720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.301982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] PATH_INFO: `/v3/users/e79e89b7261f4751ae9cd79132ff7ae2` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.303161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] RBAC: Authorizing `identity:get_user(user_id=e79e89b7261f4751ae9cd79132ff7ae2)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.304638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.310556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 33/77] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/users/e79e89b7261f4751ae9cd79132ff7ae2 => generated 247 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:18.310556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.326387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.389053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9c64e4-e4f0-4295-a1ac-2e96cd6ec9a9 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.417965 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14821c32-d339-47d8-a47e-a2786e36ecd2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.426081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.426743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.427025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.427298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.440938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.442588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.444195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 45/78] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:18.444954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.458876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.459591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.459878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.460151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.470393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:18.475153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:18.475961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 34/79] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:18.477451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.495765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.542762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12917e1-e8c2-455b-973f-ec9d3bcc6607 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.545031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.576411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.577043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.577327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.577601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.578264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:18.578596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.579970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.580407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c387066-405d-4924-99e3-a6c2856ac256 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.587771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 46/80] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:18.588569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.613761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.614362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.614669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.614938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.683698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.685173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.691198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762a3ddc-ec81-4f06-9b02-b40f60994741 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.700304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.724681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.725197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.725480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.725764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.726897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.728349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.729911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 47/81] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:18.734505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.769192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.784739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 35/82] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:18.785570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.838732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1315f3e2-5f88-412d-9c42-cc1b08ffd82d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.870963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.871564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.871847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.872121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.884958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e235e8-e309-4332-a34a-17fecf747647 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:18.899296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:18.918931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:18.919532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:18.919813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:18.920082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:18.920735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] RBAC: Adding query filter params (name=member) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:18.921038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.922417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.932512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 36/83] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:18.933561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:18.946235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:18.947726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:18.979932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.031064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd59821-5ea2-4d77-895c-39e07bf92d44 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.044176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 48/84] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:19.044997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.065289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.066511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.066794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.067064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.068198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.069638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.071231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 37/85] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:18 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:19.071987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.120232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.145754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f69dff9-d736-4d47-9c05-cc7af4c9b143 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.173694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d77f1b5-cc41-45ea-bb09-cf17df5a9c67 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.178598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.179213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.179491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.179758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.180403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.181831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.192343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.207333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.207952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.208230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.208498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.209646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.211127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.212754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 38/86] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:19 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:19.213599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.275842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 49/87] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:32:19 2022] POST /identity/v3/users => generated 289 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:19.282218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.317788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c30d31d-2198-4050-9666-9cbdcdeb8173 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.337538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.363188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.364313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.364962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.365455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.366947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.368903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.371213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 39/88] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:19 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:19.372475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.407733 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6842b62-db7f-432c-bf9f-df7f1fbe54aa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.447923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.449210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.449210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.449395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.450519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.451982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.453632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 50/89] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:19 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:19.454396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.499898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8301d4-41f2-484f-80a2-8703e05b6564 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.518456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.538442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.538775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.539066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.539342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.572837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b28b96-b373-44c1-8b8e-460c40bb4cb1 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.604068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.622523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.623061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.623343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.623962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.624339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=e79e89b7261f4751ae9cd79132ff7ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:19.624878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.626087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.632388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 51/90] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:19 2022] GET /identity/v3/role_assignments?user.id=e79e89b7261f4751ae9cd79132ff7ae2&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:19.632540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.633167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.633916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.752014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 40/91] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:19.753287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:19.789096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adc555-b467-479a-a2e6-4a3118899228 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.836780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.857423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.858052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.858338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.858639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.890788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:19.927886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5746c0d-b370-41df-9365-74154ba88fff None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:19.965235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.966626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:19.994063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:19.994890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:19.995113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:19.995358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:19.996200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=547771dc36184db8b769aa9879b27b74) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:19.996519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:19.997964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:20.009190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 41/92] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:19 2022] GET /identity/v3/role_assignments?user.id=547771dc36184db8b769aa9879b27b74&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 259 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:20.010038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:20.085961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 52/93] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:20.093672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:20.130860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:20.219997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326bbd50-62f5-4ad0-a544-26eb21982b77 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:20.288306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8664e553-2325-4506-9660-f3b45a62ed68 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:20.300855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:20.301577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:20.301857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:20.302191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] PATH_INFO: `/v3/projects/5078eaae99d0400298ac32aaeb148d06` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:20.303478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] RBAC: Authorizing `identity:get_project(project_id=5078eaae99d0400298ac32aaeb148d06)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:20.305043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:20.307052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 53/94] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:20 2022] GET /identity/v3/projects/5078eaae99d0400298ac32aaeb148d06 => generated 312 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:20.319158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:20.361181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:20.361751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:20.362078 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:20.363250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:20.364655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:20.366176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:20.368919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 42/95] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:20 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:20.375263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:20.423638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:20.546494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:20.599468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a3d45e-1168-4104-87e5-b20c93e1b99f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:20.669134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07124bd-7d87-4e1f-bdcd-9e47b8966d3b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:20.723254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b54d78-160d-4730-8a05-4cda005ef569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:20.723890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b54d78-160d-4730-8a05-4cda005ef569 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:20.724174 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b54d78-160d-4730-8a05-4cda005ef569 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:20.724447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b54d78-160d-4730-8a05-4cda005ef569 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:20.725188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b54d78-160d-4730-8a05-4cda005ef569 None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=5078eaae99d0400298ac32aaeb148d06, user.id=547771dc36184db8b769aa9879b27b74) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:20.725492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b54d78-160d-4730-8a05-4cda005ef569 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:20.726986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b54d78-160d-4730-8a05-4cda005ef569 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:20.740093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0678882-cb56-4082-a701-9b407adb3ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:20.741602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0678882-cb56-4082-a701-9b407adb3ddf None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:20.742144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0678882-cb56-4082-a701-9b407adb3ddf None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:20.742927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0678882-cb56-4082-a701-9b407adb3ddf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:20.744527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0678882-cb56-4082-a701-9b407adb3ddf None admin] RBAC: Adding query filter params (role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=547771dc36184db8b769aa9879b27b74) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:20.745086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0678882-cb56-4082-a701-9b407adb3ddf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:20.747688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0678882-cb56-4082-a701-9b407adb3ddf None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:20.748330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 54/96] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:20 2022] GET /identity/v3/role_assignments?user.id=547771dc36184db8b769aa9879b27b74&scope.project.id=5078eaae99d0400298ac32aaeb148d06&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 259 bytes in 430 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:20.767848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 43/97] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:20 2022] GET /identity/v3/role_assignments?user.id=547771dc36184db8b769aa9879b27b74&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 259 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:22.421622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20c3b28-d4e0-402f-8a41-532729c7f397 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:22.422644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20c3b28-d4e0-402f-8a41-532729c7f397 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:22.423131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20c3b28-d4e0-402f-8a41-532729c7f397 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:22.425428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 55/98] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:22.431415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d7b9b-6d71-4e0c-b4cb-0090508e93c0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:22.432257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d7b9b-6d71-4e0c-b4cb-0090508e93c0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:22.432758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d7b9b-6d71-4e0c-b4cb-0090508e93c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:22.459403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-fb5d7b9b-6d71-4e0c-b4cb-0090508e93c0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:22.460941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fb5d7b9b-6d71-4e0c-b4cb-0090508e93c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:22.548720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 44/99] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:22.842080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b79c256-542b-4c8c-81bd-ddb5e84dabeb None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:22.842806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b79c256-542b-4c8c-81bd-ddb5e84dabeb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:22.843321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b79c256-542b-4c8c-81bd-ddb5e84dabeb None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:22.844930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 56/100] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:22.859368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ce5d81-a4ab-4c63-9939-dfbb64a509d1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:22.859933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ce5d81-a4ab-4c63-9939-dfbb64a509d1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:22.860410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ce5d81-a4ab-4c63-9939-dfbb64a509d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:22.880205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:22.886531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d4ce5d81-a4ab-4c63-9939-dfbb64a509d1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:22.887979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d4ce5d81-a4ab-4c63-9939-dfbb64a509d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:22.977668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 45/101] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:23.024727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b79c256-542b-4c8c-81bd-ddb5e84dabeb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:23.190867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b79c256-542b-4c8c-81bd-ddb5e84dabeb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:23.197915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fdd907-dc69-45fa-aee4-e88a25820e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:23.198606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fdd907-dc69-45fa-aee4-e88a25820e32 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.199022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fdd907-dc69-45fa-aee4-e88a25820e32 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.199301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fdd907-dc69-45fa-aee4-e88a25820e32 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.211377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fdd907-dc69-45fa-aee4-e88a25820e32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-76fdd907-dc69-45fa-aee4-e88a25820e32 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:23.216444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:23.217298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:23.218087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 57/102] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:23.224252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ce5d81-a4ab-4c63-9939-dfbb64a509d1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:23.307697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2613b5ca-f19c-43a9-94eb-0cad09ab1445 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.308031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2613b5ca-f19c-43a9-94eb-0cad09ab1445 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.308312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2613b5ca-f19c-43a9-94eb-0cad09ab1445 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.309665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 58/103] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:23.314722 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cc8a5-eee3-4fcc-b2ab-3ba8fccb94b0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.315004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cc8a5-eee3-4fcc-b2ab-3ba8fccb94b0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.315837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cc8a5-eee3-4fcc-b2ab-3ba8fccb94b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.342498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-4d6cc8a5-eee3-4fcc-b2ab-3ba8fccb94b0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:23.343716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4d6cc8a5-eee3-4fcc-b2ab-3ba8fccb94b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:23.361182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:23.366608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ce5d81-a4ab-4c63-9939-dfbb64a509d1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:23.430441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:23.431071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.431355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.431631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.432308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] RBAC: Adding query filter params (name=reader) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:23.432736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:23.434351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:23.436770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:23.452551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 46/104] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:23.452551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:23.545109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 59/105] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:23.546722 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6cc8a5-eee3-4fcc-b2ab-3ba8fccb94b0 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:23.566016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7d206-b0b7-441b-8ba3-ffbce37ca6ab None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:23.603832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dcddc3-4bb5-45a8-9564-4a04e9d029d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:23.604482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dcddc3-4bb5-45a8-9564-4a04e9d029d8 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.604767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dcddc3-4bb5-45a8-9564-4a04e9d029d8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.605045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dcddc3-4bb5-45a8-9564-4a04e9d029d8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.606196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dcddc3-4bb5-45a8-9564-4a04e9d029d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:23.607677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dcddc3-4bb5-45a8-9564-4a04e9d029d8 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:23.614637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 47/106] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:23.614637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e868c737-180b-4b51-a248-8a1da63d5eec None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.614637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e868c737-180b-4b51-a248-8a1da63d5eec None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.614637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e868c737-180b-4b51-a248-8a1da63d5eec None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.614637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 48/107] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:23 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:23.618382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feaedf8-c805-4a14-8ea7-6d389d447255 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.618695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feaedf8-c805-4a14-8ea7-6d389d447255 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.618977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feaedf8-c805-4a14-8ea7-6d389d447255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.637942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-3feaedf8-c805-4a14-8ea7-6d389d447255 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:23.639296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3feaedf8-c805-4a14-8ea7-6d389d447255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:23.671526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6cc8a5-eee3-4fcc-b2ab-3ba8fccb94b0 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:23.708404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:23.708977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.709258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.709533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] PATH_INFO: `/v3/users/4cbc7a20c3cb45d388be02cb597f6703` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.723606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] RBAC: Authorizing `identity:get_user(user_id=4cbc7a20c3cb45d388be02cb597f6703)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:23.725433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:23.727305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 60/108] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:23 2022] GET /identity/v3/users/4cbc7a20c3cb45d388be02cb597f6703 => generated 289 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:23.728159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:23.739674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:23.748379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:23.756094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 49/109] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:23.763936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a491a75-2bda-4f21-a350-052aaf935d33 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.763936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a491a75-2bda-4f21-a350-052aaf935d33 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.763936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a491a75-2bda-4f21-a350-052aaf935d33 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.763936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 50/110] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:23.765034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a491a75-2bda-4f21-a350-052aaf935d33 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:23.849555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a9de3-a274-4d39-82d1-0f5213449452 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:23.895749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884c3cac-3f16-43a5-90c2-d2a65b3b2daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:23.896416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884c3cac-3f16-43a5-90c2-d2a65b3b2daf None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.896694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884c3cac-3f16-43a5-90c2-d2a65b3b2daf None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.896998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884c3cac-3f16-43a5-90c2-d2a65b3b2daf None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:23.897657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884c3cac-3f16-43a5-90c2-d2a65b3b2daf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:23.900812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884c3cac-3f16-43a5-90c2-d2a65b3b2daf None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:23.932424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a491a75-2bda-4f21-a350-052aaf935d33 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:23.969034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:23.969556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:23.969831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:23.970100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.024007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 61/111] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:32:23 2022] POST /identity/v3/projects => generated 302 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:24.029448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e445e5ee-7010-4206-8016-6e9ca6ca3c97 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.029448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e445e5ee-7010-4206-8016-6e9ca6ca3c97 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.029448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e445e5ee-7010-4206-8016-6e9ca6ca3c97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.052203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e445e5ee-7010-4206-8016-6e9ca6ca3c97 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:24.054523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:24.055260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e445e5ee-7010-4206-8016-6e9ca6ca3c97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:24.066459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:24.068610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:24.127015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:24.150293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 62/112] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:24.151662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e445e5ee-7010-4206-8016-6e9ca6ca3c97 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.200264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 51/113] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:24.201146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.279403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e445e5ee-7010-4206-8016-6e9ca6ca3c97 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.310840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:24.317469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d624ff81-53d6-4ec5-bcea-67a731f74d6f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.324904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.325809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.326286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.326805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.340024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:24.345166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:24.345990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:24.346808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:24.349176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 63/114] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:24.350299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.353463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.354194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.354567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.354912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.366203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:24.371676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:24.372689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:24.373602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:24.382053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 52/115] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:24.382776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.443440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:24.493777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d1599b-c206-4e34-8177-299d2f5ff41d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.532638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b72b2bb-fd2a-4cda-a48a-2946ac8a45a2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.533348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.535081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.535565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.535846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.536538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:24.536848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:24.538230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:24.547210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 53/116] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:24.548244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.567751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:24.576480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.577111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.577387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.577654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.578926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:24.580389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:24.582745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 64/117] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:24.583344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.717417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafaf1a9-a4d9-4cc6-a07e-8cafdd2cca81 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.718038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec80e27-0585-4963-a306-9e3aae70a818 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.736783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:24.759810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebf513e-bc5d-4295-bde4-faaf4c20c4bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.760786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebf513e-bc5d-4295-bde4-faaf4c20c4bf None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.761278 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebf513e-bc5d-4295-bde4-faaf4c20c4bf None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.761762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebf513e-bc5d-4295-bde4-faaf4c20c4bf None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.762030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.762704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.762980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.763250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] PATH_INFO: `/v3/roles/member` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.763532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebf513e-bc5d-4295-bde4-faaf4c20c4bf None admin] RBAC: Adding query filter params (name=member) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:24.764037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebf513e-bc5d-4295-bde4-faaf4c20c4bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:24.765624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebf513e-bc5d-4295-bde4-faaf4c20c4bf None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:24.774745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:24.776016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 54/118] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:24.777363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0184d820-0cf4-4799-aa38-602064b0fa7d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0184d820-0cf4-4799-aa38-602064b0fa7d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.778225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0184d820-0cf4-4799-aa38-602064b0fa7d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.779613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 55/119] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:24.780151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:24.780954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:24.781699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 65/120] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:24.782309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.782907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0184d820-0cf4-4799-aa38-602064b0fa7d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.873388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:24.891256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d2509-da7f-431e-ab14-75999e9134ab None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.897675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0184d820-0cf4-4799-aa38-602064b0fa7d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:24.928084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.928701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.928983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.929257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.930019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] RBAC: Adding query filter params (role.id=35cf5ed465b441dc825c683f9e9a02a6, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=4cbc7a20c3cb45d388be02cb597f6703) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:24.930296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:24.931791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:24.933428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af134079-f63d-41b6-959f-105b1e201298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:24.934064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af134079-f63d-41b6-959f-105b1e201298 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.934341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af134079-f63d-41b6-959f-105b1e201298 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.934644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af134079-f63d-41b6-959f-105b1e201298 None admin] PATH_INFO: `/v3/users/e79e89b7261f4751ae9cd79132ff7ae2` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.935767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af134079-f63d-41b6-959f-105b1e201298 None admin] RBAC: Authorizing `identity:get_user(user_id=e79e89b7261f4751ae9cd79132ff7ae2)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:24.937196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af134079-f63d-41b6-959f-105b1e201298 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:24.942726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 66/121] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/role_assignments?user.id=4cbc7a20c3cb45d388be02cb597f6703&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=35cf5ed465b441dc825c683f9e9a02a6 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:24.942726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:24.942976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 56/122] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/users/e79e89b7261f4751ae9cd79132ff7ae2 => generated 247 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:24.943847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684857e9-8500-4426-939f-d6e543101074 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:24.944131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684857e9-8500-4426-939f-d6e543101074 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:24.944399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684857e9-8500-4426-939f-d6e543101074 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:24.963338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-684857e9-8500-4426-939f-d6e543101074 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:24.964713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-684857e9-8500-4426-939f-d6e543101074 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:24.999473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:25.045979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 57/123] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:25.047301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684857e9-8500-4426-939f-d6e543101074 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:25.130070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b16c9f-fef4-4d37-b315-0bb00b83714f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.138431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:25.161586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684857e9-8500-4426-939f-d6e543101074 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.193135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.194043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.194332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.194716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.196055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:25.198210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:25.200115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 67/124] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:24 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:25.200688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:25.210241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.211243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.212453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.212927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.213882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] RBAC: Adding query filter params (name=member) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:25.216729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:25.218372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:25.236345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 58/125] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:25.238884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:25.410927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3b676d-d80c-4d88-b4ae-b06797b95a04 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.415179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:25.438073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10885a1-2f82-44cd-9c40-6a65a2fbf5f1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.451345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.452242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.452684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.453028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.454291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:25.456147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:25.458173 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 68/126] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:25 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:25.459149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:25.488338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.488978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.489302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.489575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] PATH_INFO: `/v3/projects/5078eaae99d0400298ac32aaeb148d06` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.491005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] RBAC: Authorizing `identity:get_project(project_id=5078eaae99d0400298ac32aaeb148d06)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:25.493255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:25.500274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 59/127] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:25 2022] GET /identity/v3/projects/5078eaae99d0400298ac32aaeb148d06 => generated 312 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:25.500274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:25.511854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:25.593276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4bb470-0abe-4115-98ee-0ac40e8d7f1b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.623818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c3458b-af4d-43d9-958d-e28da742721c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.632751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.632751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.632751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.633761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.638662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:25.638662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:25.638662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 69/128] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:25 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:25.639633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:25.666154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.666759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.667042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.667584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.685791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:25.693203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:25.697693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:25.699408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 60/129] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:32:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:25.701177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:25.763841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeebb9c-95b4-4f89-b958-dbb530891328 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.782614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:25.804193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.804869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.805151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.805432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/e79e89b7261f4751ae9cd79132ff7ae2/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.807471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] RBAC: Authorizing `identity:create_grant(project_id=7536b89e67c444c584e6eb0afdb7c343, user_id=e79e89b7261f4751ae9cd79132ff7ae2, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:25.808922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:25.820894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da871d10-c4a6-4fcb-8b4e-59cb9b39f43e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:25.857375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:25.858034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:25.858310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:25.858614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] PATH_INFO: `/v3/projects/5078eaae99d0400298ac32aaeb148d06/users/547771dc36184db8b769aa9879b27b74/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:25.860496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] RBAC: Authorizing `identity:create_grant(project_id=5078eaae99d0400298ac32aaeb148d06, user_id=547771dc36184db8b769aa9879b27b74, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:25.862557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:25.895141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 70/130] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:25 2022] PUT /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/e79e89b7261f4751ae9cd79132ff7ae2/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:25.896058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:26.013173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 61/131] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:25 2022] PUT /identity/v3/projects/5078eaae99d0400298ac32aaeb148d06/users/547771dc36184db8b769aa9879b27b74/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:26.014214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:26.054120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:26.062020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aa088c-e7fa-4012-af00-109d2d18bb39 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:26.136629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:26.137615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.138076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.138544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.140820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:26.143412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:26.145506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 71/132] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:25 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:26.151081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:26.211825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc40939-13a6-476b-a6a4-043342aa1337 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:26.248755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:26.260157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:26.261222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.261663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.262249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.264175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:26.264175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:26.265570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:26.275532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9b01bd-3149-48c6-978f-325b9d8af1d1 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:26.280461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 62/133] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:32:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:26.284048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:26.321022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:26.321719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.321998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.322278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/547771dc36184db8b769aa9879b27b74/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.324433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] RBAC: Authorizing `identity:create_grant(project_id=7536b89e67c444c584e6eb0afdb7c343, user_id=547771dc36184db8b769aa9879b27b74, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:26.325990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:26.406536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae29ec-63ba-4b43-b3c1-70f99c5ac306 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:26.418614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 72/134] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:26 2022] PUT /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/547771dc36184db8b769aa9879b27b74/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:26.449593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:26.450255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.450567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.450837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.452191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:26.453792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:26.457859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 63/135] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:26 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:26.462055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:26.499106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:26.596219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393ded0-7973-4e2c-957b-0fdc5037c639 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:26.640930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21fbfa2-c573-4c83-8fdf-89ec52af8097 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:26.641625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fbfa2-c573-4c83-8fdf-89ec52af8097 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.641912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fbfa2-c573-4c83-8fdf-89ec52af8097 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.642194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fbfa2-c573-4c83-8fdf-89ec52af8097 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.644393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21fbfa2-c573-4c83-8fdf-89ec52af8097 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:26.645990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21fbfa2-c573-4c83-8fdf-89ec52af8097 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:26.647661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 73/136] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:26 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:26.653488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:26.725405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:26.772601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e17811-6d57-474f-be3a-7dc2f215c016 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.772889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e17811-6d57-474f-be3a-7dc2f215c016 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.773151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e17811-6d57-474f-be3a-7dc2f215c016 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.774563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 74/137] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:26.775170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c18e5c8-b994-4166-99ea-90d2477ef397 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:26.779562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fb90fa-05bb-409b-b87f-c92a901b361a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.779840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fb90fa-05bb-409b-b87f-c92a901b361a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.780115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fb90fa-05bb-409b-b87f-c92a901b361a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.805715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b8fb90fa-05bb-409b-b87f-c92a901b361a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:26.807439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b8fb90fa-05bb-409b-b87f-c92a901b361a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:26.816200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:26.816885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:26.817161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:26.817434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/547771dc36184db8b769aa9879b27b74/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:26.824523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=7536b89e67c444c584e6eb0afdb7c343, user_id=547771dc36184db8b769aa9879b27b74, role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:26.825982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:26.884909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 75/138] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:26.930533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 64/139] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:26 2022] PUT /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/547771dc36184db8b769aa9879b27b74/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:27.264396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fb90fa-05bb-409b-b87f-c92a901b361a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:27.296821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:27.402232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fb90fa-05bb-409b-b87f-c92a901b361a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:27.445335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed728c40-a178-4470-bd1c-5297546e29e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:27.446323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed728c40-a178-4470-bd1c-5297546e29e2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:27.446753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed728c40-a178-4470-bd1c-5297546e29e2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:27.447183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed728c40-a178-4470-bd1c-5297546e29e2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:27.448433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed728c40-a178-4470-bd1c-5297546e29e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:27.450040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed728c40-a178-4470-bd1c-5297546e29e2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:27.451787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 76/140] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:27.458102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:27.609749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf7487-7567-40e1-962a-16e4636357b8 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:27.629138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:27.646787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:27.647466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:27.647754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:27.648022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:27.648687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:27.650193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:27.686253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f4290-04c3-4d64-a2b5-d4ffc4bb52db None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:27.687250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f4290-04c3-4d64-a2b5-d4ffc4bb52db None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:27.687731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f4290-04c3-4d64-a2b5-d4ffc4bb52db None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:27.689271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 77/141] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:27.694687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130e8fa3-52eb-4c68-b587-c01fa57dbdb0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:27.695199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130e8fa3-52eb-4c68-b587-c01fa57dbdb0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:27.695665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130e8fa3-52eb-4c68-b587-c01fa57dbdb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:27.714274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-130e8fa3-52eb-4c68-b587-c01fa57dbdb0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:27.715809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-130e8fa3-52eb-4c68-b587-c01fa57dbdb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:27.774510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 65/142] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:32:27 2022] POST /identity/v3/users => generated 283 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:27.809955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 78/143] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:28.109801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:28.243205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a0b73-10a2-49b0-bf64-b96821563e2d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:28.279304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:28.279944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:28.280223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:28.280492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:28.295721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:28.302939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:28.303791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:28.304988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 66/144] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:28.312864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130e8fa3-52eb-4c68-b587-c01fa57dbdb0 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:28.357932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:28.487294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130e8fa3-52eb-4c68-b587-c01fa57dbdb0 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:28.533597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:28.534779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:28.535064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:28.535339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:28.536012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:28.536313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:28.537698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:28.550710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 79/145] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:28.556408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:28.611555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:28.705294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b31e20-049f-4359-b6c5-25086955b28e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:28.750533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:28.751178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:28.751461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:28.751737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] PATH_INFO: `/v3/users/4cbc7a20c3cb45d388be02cb597f6703` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:28.759195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] RBAC: Authorizing `identity:get_user(user_id=4cbc7a20c3cb45d388be02cb597f6703)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:28.760690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:28.762574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 67/146] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:28 2022] GET /identity/v3/users/4cbc7a20c3cb45d388be02cb597f6703 => generated 289 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:28.768612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:28.873776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:28.890470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673fcfb9-a94e-4a13-b475-7265161deb43 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:28.933231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f52d38d-101c-4cd2-95ba-660a90a86f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:28.933873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52d38d-101c-4cd2-95ba-660a90a86f50 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:28.934157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52d38d-101c-4cd2-95ba-660a90a86f50 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:28.934447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52d38d-101c-4cd2-95ba-660a90a86f50 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:28.935619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f52d38d-101c-4cd2-95ba-660a90a86f50 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:28.937075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f52d38d-101c-4cd2-95ba-660a90a86f50 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:28.939060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 80/147] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:28 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:28.945633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:29.073991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf376af-0efb-4d64-b69c-a0213a2ccf49 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:29.075676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:29.117592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2762bd-33ea-4af4-bb3d-f60db7b12e62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:29.118568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2762bd-33ea-4af4-bb3d-f60db7b12e62 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:29.118862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2762bd-33ea-4af4-bb3d-f60db7b12e62 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:29.119150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2762bd-33ea-4af4-bb3d-f60db7b12e62 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/4cbc7a20c3cb45d388be02cb597f6703/roles/35cf5ed465b441dc825c683f9e9a02a6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:29.121203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2762bd-33ea-4af4-bb3d-f60db7b12e62 None admin] RBAC: Authorizing `identity:create_grant(project_id=7536b89e67c444c584e6eb0afdb7c343, user_id=4cbc7a20c3cb45d388be02cb597f6703, role_id=35cf5ed465b441dc825c683f9e9a02a6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:29.122937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2762bd-33ea-4af4-bb3d-f60db7b12e62 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:29.197172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 68/148] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:28 2022] PUT /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343/users/4cbc7a20c3cb45d388be02cb597f6703/roles/35cf5ed465b441dc825c683f9e9a02a6 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:29.245049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66edcd-0028-4cf9-a36d-0a6d516281cb None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:29.245333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66edcd-0028-4cf9-a36d-0a6d516281cb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:29.245603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66edcd-0028-4cf9-a36d-0a6d516281cb None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:29.247175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 81/149] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:29.255922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96807bfb-efb7-4ed8-80e4-b91f330e7800 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:29.256565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96807bfb-efb7-4ed8-80e4-b91f330e7800 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:29.257039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96807bfb-efb7-4ed8-80e4-b91f330e7800 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:29.282832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-96807bfb-efb7-4ed8-80e4-b91f330e7800 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:29.285229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-96807bfb-efb7-4ed8-80e4-b91f330e7800 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:29.392968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 69/150] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:29.642687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ec222-8b36-40f4-afd8-17774c856af3 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:29.643013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ec222-8b36-40f4-afd8-17774c856af3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:29.643747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ec222-8b36-40f4-afd8-17774c856af3 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:29.645685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 82/151] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:29.651771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3add6dd-9810-4f24-9416-0b279f25c327 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:29.652057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3add6dd-9810-4f24-9416-0b279f25c327 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:29.652329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3add6dd-9810-4f24-9416-0b279f25c327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:29.676133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e3add6dd-9810-4f24-9416-0b279f25c327 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:29.677351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e3add6dd-9810-4f24-9416-0b279f25c327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:29.757685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 70/152] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:29.871825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33ec222-8b36-40f4-afd8-17774c856af3 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:30.012093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33ec222-8b36-40f4-afd8-17774c856af3 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:30.057155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6f7186-ef4b-4b23-8428-e42f8f2f1aa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:30.058176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6f7186-ef4b-4b23-8428-e42f8f2f1aa6 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.058668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6f7186-ef4b-4b23-8428-e42f8f2f1aa6 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.059145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6f7186-ef4b-4b23-8428-e42f8f2f1aa6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.073115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6f7186-ef4b-4b23-8428-e42f8f2f1aa6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-de6f7186-ef4b-4b23-8428-e42f8f2f1aa6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:30.078923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:30.079718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:30.080475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:30.082788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 83/153] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:30.091531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3add6dd-9810-4f24-9416-0b279f25c327 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:30.141448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:30.229426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79465bd-3cd0-46b7-bfa0-18ce92329815 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.230040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79465bd-3cd0-46b7-bfa0-18ce92329815 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.230567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79465bd-3cd0-46b7-bfa0-18ce92329815 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.232540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 84/154] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:30.238366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a2176b-960c-43c2-8aa6-8eef264a9216 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.238911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a2176b-960c-43c2-8aa6-8eef264a9216 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.239411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a2176b-960c-43c2-8aa6-8eef264a9216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.239810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3add6dd-9810-4f24-9416-0b279f25c327 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:30.267648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-89a2176b-960c-43c2-8aa6-8eef264a9216 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:30.270089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-89a2176b-960c-43c2-8aa6-8eef264a9216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:30.286373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:30.320423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:30.321059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.321342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.321616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.322304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:30.322741 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:30.324488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:30.349916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 71/155] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:30.350802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:30.377441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 85/156] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:30.379017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a2176b-960c-43c2-8aa6-8eef264a9216 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:30.545174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:30.571192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3ab213-ebe6-4f45-9f76-35a5ff7e84fd None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:30.572598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a2176b-960c-43c2-8aa6-8eef264a9216 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:30.623736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:30.624389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.624671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.624939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.643946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:30.645771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:30.646686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.647157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.647611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] PATH_INFO: `/v3/users/e79e89b7261f4751ae9cd79132ff7ae2` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.648928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] RBAC: Authorizing `identity:get_user(user_id=e79e89b7261f4751ae9cd79132ff7ae2)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:30.649046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:30.649838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:30.650599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 72/157] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:30.653576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:30.660184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 86/158] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:30 2022] GET /identity/v3/users/e79e89b7261f4751ae9cd79132ff7ae2 => generated 247 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:30.662667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:30.662798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:30.698439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:30.821647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b0f88-7ccc-4182-a51f-fd0bc9b7304b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:30.823374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b993aa09-6557-4277-98b3-a1f2286b5391 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:30.879744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae73178-11f0-4ab9-b859-49818859ccae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:30.880366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae73178-11f0-4ab9-b859-49818859ccae None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.880645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae73178-11f0-4ab9-b859-49818859ccae None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.880913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae73178-11f0-4ab9-b859-49818859ccae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.890131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:30.891403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:30.891935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:30.892488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:30.893473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:30.893964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:30.895685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:30.903127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:30.916396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 87/159] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:30.922330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:30.978037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae73178-11f0-4ab9-b859-49818859ccae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:30.978037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae73178-11f0-4ab9-b859-49818859ccae None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:31.064104 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:31.135739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1a0de0-8ef0-4c34-a59c-8982764b3b66 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:31.168615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 73/160] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:31.169902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a7a424-5b68-4d70-983a-188dcc1a8abb None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:31.170188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a7a424-5b68-4d70-983a-188dcc1a8abb None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:31.170488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a7a424-5b68-4d70-983a-188dcc1a8abb None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:31.171737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 74/161] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:31.172434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a7a424-5b68-4d70-983a-188dcc1a8abb None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:31.215143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb279d64-e73f-4c16-a2e7-f9771fc0f61e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:31.215775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb279d64-e73f-4c16-a2e7-f9771fc0f61e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:31.216053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb279d64-e73f-4c16-a2e7-f9771fc0f61e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:31.216326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb279d64-e73f-4c16-a2e7-f9771fc0f61e None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:31.217491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb279d64-e73f-4c16-a2e7-f9771fc0f61e None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:31.219139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb279d64-e73f-4c16-a2e7-f9771fc0f61e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:31.220922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 88/162] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:30 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:31.222084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9beb6e-6f93-4abe-9aca-4888d9d269dc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:31.222366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9beb6e-6f93-4abe-9aca-4888d9d269dc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:31.222656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9beb6e-6f93-4abe-9aca-4888d9d269dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:31.238768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:31.256526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ef9beb6e-6f93-4abe-9aca-4888d9d269dc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:31.257800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ef9beb6e-6f93-4abe-9aca-4888d9d269dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:31.356800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a7a424-5b68-4d70-983a-188dcc1a8abb None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:31.412808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 89/163] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:31.413675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9beb6e-6f93-4abe-9aca-4888d9d269dc None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:31.452151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:31.452846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:31.453119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:31.453421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:31.483980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:31.500242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:31.501110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:31.501857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 75/164] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:31.502534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:31.582771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:31.645859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9beb6e-6f93-4abe-9aca-4888d9d269dc None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:31.703358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:31.703923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:31.704444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:31.704893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:31.706282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:31.707980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:31.710088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 90/165] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:31 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:31.711016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:31.752583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7208bdda-6fee-4f29-86c7-fa2a98fb5888 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:31.760825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:31.831650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:31.832269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:31.832543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:31.832807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:31.921158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ac43fd-012f-4657-bffb-e11bd7dbb852 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:31.967138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:31.968527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:31.992617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:31.993498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:31.993777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:31.994046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:31.994749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:31.995049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:31.996444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:32.024075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 91/166] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:32.024935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:32.031005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:32.202727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 76/167] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 701 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:32.203594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:32.210732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:32.233271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210b42a7-cc15-4f94-81dd-440a331a5510 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:32.297106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:32.297718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:32.297996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:32.298265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:32.299103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=e79e89b7261f4751ae9cd79132ff7ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:32.299399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:32.302922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:32.314082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 92/168] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:32 2022] GET /identity/v3/role_assignments?user.id=e79e89b7261f4751ae9cd79132ff7ae2&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:32.315019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:32.435721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e2336c-582f-497d-9424-585a091d9f1b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:32.479716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:32.515916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:32.516486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:32.516764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:32.517034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:32.543587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:32.548572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:32.550085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:32.551313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 77/169] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:32:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 347 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:32.551944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:32.589482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe347c-29e1-480f-9d60-25426d5de37a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:32.695242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:32.695922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:32.696029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:32.696206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:32.696482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:32.697793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:32.699279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:32.700935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 93/170] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:32 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:32.701720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:32.763592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a53355-83d7-4745-bda2-d5b9d08ec370 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:32.827356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758c24c5-bd9b-4c64-994f-ea96aa762477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:32.827918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c24c5-bd9b-4c64-994f-ea96aa762477 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:32.828193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c24c5-bd9b-4c64-994f-ea96aa762477 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:32.828461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c24c5-bd9b-4c64-994f-ea96aa762477 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:32.830386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758c24c5-bd9b-4c64-994f-ea96aa762477 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:32.832438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758c24c5-bd9b-4c64-994f-ea96aa762477 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:32.834076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 78/171] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:32 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:32.835230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab865db6-ae45-464a-ba2c-7289cddb8736 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:32.835510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab865db6-ae45-464a-ba2c-7289cddb8736 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:32.835781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab865db6-ae45-464a-ba2c-7289cddb8736 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:32.840902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 79/172] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:32.844984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab865db6-ae45-464a-ba2c-7289cddb8736 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:32.899384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c467f8-ec4c-403b-b457-43c32559ffe7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:32.964113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:32.964764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:32.965011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:32.965278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:32.965964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:32.966267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:32.967725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:32.972583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:32.990826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 94/173] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:32:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:32.991788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:33.048040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab865db6-ae45-464a-ba2c-7289cddb8736 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:33.108893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:33.115978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:33.116744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.121961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.121961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.173515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd53f9f-5a15-408a-8ba3-a5147c90e1a2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:33.214859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010c1732-56aa-4270-adca-7ec30acc3984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:33.215389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:33.215831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010c1732-56aa-4270-adca-7ec30acc3984 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.215987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010c1732-56aa-4270-adca-7ec30acc3984 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.216248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010c1732-56aa-4270-adca-7ec30acc3984 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.216870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:33.216960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010c1732-56aa-4270-adca-7ec30acc3984 None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=547771dc36184db8b769aa9879b27b74) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:33.217261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010c1732-56aa-4270-adca-7ec30acc3984 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:33.218793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010c1732-56aa-4270-adca-7ec30acc3984 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:33.225307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 95/174] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:32 2022] GET /identity/v3/role_assignments?user.id=547771dc36184db8b769aa9879b27b74&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 608 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:33.226541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7967f1-9408-4870-a6fd-4e0c8bc025ea None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.226751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7967f1-9408-4870-a6fd-4e0c8bc025ea None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.229662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7967f1-9408-4870-a6fd-4e0c8bc025ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.249309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0b7967f1-9408-4870-a6fd-4e0c8bc025ea None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:33.251270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0b7967f1-9408-4870-a6fd-4e0c8bc025ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:33.330343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:33.336260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 80/175] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:33.337929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:33.343221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 96/176] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:33.344781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a36dc-713c-4b76-9842-233a449a2c8b None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.351162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a36dc-713c-4b76-9842-233a449a2c8b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.351162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a36dc-713c-4b76-9842-233a449a2c8b None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.351162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 97/177] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:33 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:33.352300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ff5dc2-76ce-4572-b1ab-d57ae9b254b7 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.352826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ff5dc2-76ce-4572-b1ab-d57ae9b254b7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.353309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ff5dc2-76ce-4572-b1ab-d57ae9b254b7 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.354759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 98/178] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:33.356122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105b2d5f-8712-4921-ae4a-fff5ab259422 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.356671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105b2d5f-8712-4921-ae4a-fff5ab259422 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.357136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105b2d5f-8712-4921-ae4a-fff5ab259422 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.358652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 99/179] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:33.359667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105b2d5f-8712-4921-ae4a-fff5ab259422 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:33.396155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:33.485676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fa35ba-6c5c-4fd4-9185-fee0157dfb6e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:33.509684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105b2d5f-8712-4921-ae4a-fff5ab259422 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:33.530276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4dfd1e1-72ab-4d44-b098-f67a3b18dd4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:33.531192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dfd1e1-72ab-4d44-b098-f67a3b18dd4e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.531610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dfd1e1-72ab-4d44-b098-f67a3b18dd4e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.532016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dfd1e1-72ab-4d44-b098-f67a3b18dd4e None admin] PATH_INFO: `/v3/users/547771dc36184db8b769aa9879b27b74` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.533482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4dfd1e1-72ab-4d44-b098-f67a3b18dd4e None admin] RBAC: Authorizing `identity:get_user(user_id=547771dc36184db8b769aa9879b27b74)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:33.535341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4dfd1e1-72ab-4d44-b098-f67a3b18dd4e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:33.537133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 81/180] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:33 2022] GET /identity/v3/users/547771dc36184db8b769aa9879b27b74 => generated 275 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:33.538491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae247ae-eee5-47c6-b3f8-73761221c9a8 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.538968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae247ae-eee5-47c6-b3f8-73761221c9a8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.539426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae247ae-eee5-47c6-b3f8-73761221c9a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.552879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164fef85-ddbf-4610-82e6-dc856119af15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:33.553524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164fef85-ddbf-4610-82e6-dc856119af15 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.553805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164fef85-ddbf-4610-82e6-dc856119af15 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.554075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164fef85-ddbf-4610-82e6-dc856119af15 None admin] PATH_INFO: `/v3/projects/5078eaae99d0400298ac32aaeb148d06` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.555242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164fef85-ddbf-4610-82e6-dc856119af15 None admin] RBAC: Authorizing `identity:get_project(project_id=5078eaae99d0400298ac32aaeb148d06)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:33.556687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164fef85-ddbf-4610-82e6-dc856119af15 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:33.558305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 100/181] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:33 2022] GET /identity/v3/projects/5078eaae99d0400298ac32aaeb148d06 => generated 312 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:33.559395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-cae247ae-eee5-47c6-b3f8-73761221c9a8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:33.559568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5b691-abd9-4152-b97f-a9b24cb1d851 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.559850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5b691-abd9-4152-b97f-a9b24cb1d851 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.560132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5b691-abd9-4152-b97f-a9b24cb1d851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.561316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cae247ae-eee5-47c6-b3f8-73761221c9a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:33.578080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b7a5b691-abd9-4152-b97f-a9b24cb1d851 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:33.579285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b7a5b691-abd9-4152-b97f-a9b24cb1d851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:33.633115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 82/182] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:33.635155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:33.636208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db9ee5-bc07-462f-81e6-b98e340cb504 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.636649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db9ee5-bc07-462f-81e6-b98e340cb504 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.636933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0db9ee5-bc07-462f-81e6-b98e340cb504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.649523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 101/183] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:33.656130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a0db9ee5-bc07-462f-81e6-b98e340cb504 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:33.657321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a0db9ee5-bc07-462f-81e6-b98e340cb504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:33.662047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5b691-abd9-4152-b97f-a9b24cb1d851 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:33.683462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:33.733390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 83/184] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:33.741447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db9ee5-bc07-462f-81e6-b98e340cb504 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:33.778486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5b691-abd9-4152-b97f-a9b24cb1d851 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:33.830333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:33.831989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.832461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.832996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.891840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:33.906784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:33.908623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:33.910226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0db9ee5-bc07-462f-81e6-b98e340cb504 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:33.945641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:33.946531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:33.946946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:33.947354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:33.948231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=5078eaae99d0400298ac32aaeb148d06, user.id=547771dc36184db8b769aa9879b27b74) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:33.948675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:33.950260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:33.956250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 84/185] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:33 2022] GET /identity/v3/role_assignments?user.id=547771dc36184db8b769aa9879b27b74&scope.project.id=5078eaae99d0400298ac32aaeb148d06&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 608 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:33.957032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:33.994985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.032404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 102/186] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:34.035551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.098987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da06ccb7-7547-44d1-910e-64a1b698c2ad None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.146577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.148487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.149025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.149296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.149567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.161366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:34.161828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313b64e-70ca-4093-82f6-337f5e2eecb7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:34.167284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:34.168141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:34.169017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 85/187] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:34.169640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.208853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.209504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.209789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.210060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.224502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:34.230885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:34.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:34.232880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:34.234717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 103/188] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:34.235776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.263989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.280301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9826560b-ad81-4ffa-9631-add75e167854 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.314303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.315014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.315279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.315540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.316635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:34.318069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:34.319711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 86/189] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:34.322683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.342203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fec4ad4-455c-423f-b893-6970cae43f65 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.377421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.378035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.378301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.378594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.379776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:34.381384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:34.383061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 104/190] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:34.383827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.406389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.428806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a538046f-02fc-4dbe-bec5-44b0ef8a263e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.462313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.462969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.463245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.463515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.474141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:34.478930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:34.479813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:34.480577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 87/191] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:34.481209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.491847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e01b4f-cafe-43cf-bd56-408e430ee3fa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.495942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.524391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.525121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.525402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.525670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.526328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:34.526659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:34.528025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:34.536781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 105/192] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:34.541765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.581866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372488ba-6ad3-4a84-bc5a-87324b28c3a3 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.615872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.616469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.616742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.617014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.617118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.617663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:34.618207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:34.619613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:34.627699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 88/193] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:34.628612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.640627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c91367f-5c34-4b62-b98f-73bc96700c9b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.673538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.674071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.674966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.675485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.675844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:34.677264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:34.703317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.727308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49219770-1560-4ed2-ae38-8fdaabe7d4d2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.760776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.761375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.761644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.761906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.762819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] RBAC: Adding query filter params (role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=547771dc36184db8b769aa9879b27b74) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:34.763121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:34.764549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:34.770033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 89/194] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/role_assignments?user.id=547771dc36184db8b769aa9879b27b74&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:34.770815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.822461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 106/195] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:32:34 2022] POST /identity/v3/users => generated 297 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:34.822811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.901927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f976103f-02c7-406c-8bc2-cc1cbf0848c6 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:34.957755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:34.959066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:34.959534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:34.960548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:34.961735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:34.962262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:34.963872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:34.970378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:34.976041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 90/196] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:34.976921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:34.983879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba9f104-767a-447d-9fdc-c117c7f6c1fa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.029327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.030521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.030521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.030521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] PATH_INFO: `/v3/users/693ed507c7d3486e8bd7fc299037cd95` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.033914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.048400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] RBAC: Authorizing `identity:get_user(user_id=693ed507c7d3486e8bd7fc299037cd95)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.051319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.053144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 107/197] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/users/693ed507c7d3486e8bd7fc299037cd95 => generated 283 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:35.054010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:35.114480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236f29b3-2cfb-46e1-b4ec-29680999cc98 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.158331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.159574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.159961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.160238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] PATH_INFO: `/v3/users/693ed507c7d3486e8bd7fc299037cd95` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.161407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] RBAC: Authorizing `identity:get_user(user_id=693ed507c7d3486e8bd7fc299037cd95)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.162998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.167239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 91/198] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:34 2022] GET /identity/v3/users/693ed507c7d3486e8bd7fc299037cd95 => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:35.168176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:35.178206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeae9554-5206-4817-8793-8cc9be940b92 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.218073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.218669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.219004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.219281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] PATH_INFO: `/v3/users/4cbc7a20c3cb45d388be02cb597f6703` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.220475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] RBAC: Authorizing `identity:get_user(user_id=4cbc7a20c3cb45d388be02cb597f6703)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.222515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.224128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 108/199] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/users/4cbc7a20c3cb45d388be02cb597f6703 => generated 289 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:35.224892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:35.248551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.294949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.297074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5481baf0-a626-4604-b82a-fae086a6a9a2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.336812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.337595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.338192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.338597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.347575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f1b6a6-20bd-4926-8759-0eee7aaa1625 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.395888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.396514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.397134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.397134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.423904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.425861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.472554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.474171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.493509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.528770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 92/200] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:35.530019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:35.542802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.615883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 109/201] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:35.619392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:35.646623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90689fe7-e9cc-4de3-b92d-15ef1f5fc05f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.683007 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.683701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.684065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.684403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.732157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.774902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.777632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.779434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc55480-315a-4c16-9c15-4a8bef589271 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.816096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.817094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.817573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.818083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.819539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.821177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.823074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 110/202] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:35.824039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:35.854029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.892842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 93/203] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:35.895414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:35.932916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46811de-e8a9-46f0-b62e-56a98c290a80 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:35.966863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:35.969695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:35.970636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:35.971647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:35.972304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:35.973644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:35.975316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:35.977508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 111/204] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:35.983692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:36.006020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8f814-5ddd-4006-8ad8-3af23a432edc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:36.043644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:36.044332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:36.044605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:36.044881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:36.045602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=693ed507c7d3486e8bd7fc299037cd95) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:36.046144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:36.047353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:36.054155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 94/205] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/role_assignments?user.id=693ed507c7d3486e8bd7fc299037cd95&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:36.055172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:36.089380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:36.097020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f15cc-b01d-4771-9f91-f1807820825e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:36.140423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:36.141472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:36.141953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:36.142444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:36.144585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:36.146340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:36.148382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 112/206] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:35 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:36.160999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:36.181571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f7d6a-736c-4b40-9b10-954e2df59f6e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:36.214108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:36.219969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4fa777-8564-4fde-8236-8542903649e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:36.220589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4fa777-8564-4fde-8236-8542903649e3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:36.220867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4fa777-8564-4fde-8236-8542903649e3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:36.221820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4fa777-8564-4fde-8236-8542903649e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:36.222659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4fa777-8564-4fde-8236-8542903649e3 None admin] RBAC: Adding query filter params (role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=693ed507c7d3486e8bd7fc299037cd95) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:36.222969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4fa777-8564-4fde-8236-8542903649e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:36.225099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4fa777-8564-4fde-8236-8542903649e3 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:36.231347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 95/207] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:36 2022] GET /identity/v3/role_assignments?user.id=693ed507c7d3486e8bd7fc299037cd95&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 259 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:36.333144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca213138-77a6-46f1-802b-31924f24983f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:36.376941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acb4445-7679-4c96-9cc9-91069e809e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:36.377741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb4445-7679-4c96-9cc9-91069e809e5d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:36.378044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb4445-7679-4c96-9cc9-91069e809e5d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:36.378509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb4445-7679-4c96-9cc9-91069e809e5d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:36.382944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb4445-7679-4c96-9cc9-91069e809e5d None admin] RBAC: Adding query filter params (role.id=35cf5ed465b441dc825c683f9e9a02a6, scope.project.id=7536b89e67c444c584e6eb0afdb7c343, user.id=4cbc7a20c3cb45d388be02cb597f6703) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:36.382944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb4445-7679-4c96-9cc9-91069e809e5d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:36.382944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb4445-7679-4c96-9cc9-91069e809e5d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:36.389598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 113/208] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:36 2022] GET /identity/v3/role_assignments?user.id=4cbc7a20c3cb45d388be02cb597f6703&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=35cf5ed465b441dc825c683f9e9a02a6 => generated 608 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:37.242027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0803122-6f62-4906-8640-4588dcbd6b0d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:37.242571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0803122-6f62-4906-8640-4588dcbd6b0d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:37.242571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0803122-6f62-4906-8640-4588dcbd6b0d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:37.244524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 96/209] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:37.250173 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757097b3-1ab1-433e-a7cb-0f8771c1d007 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:37.250607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757097b3-1ab1-433e-a7cb-0f8771c1d007 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:37.250787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757097b3-1ab1-433e-a7cb-0f8771c1d007 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:37.270201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-757097b3-1ab1-433e-a7cb-0f8771c1d007 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:37.271790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-757097b3-1ab1-433e-a7cb-0f8771c1d007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:37.296693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:37.345669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 114/210] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:37.489171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c650bca-d3a9-495a-b749-30ebeef39adb None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:37.489441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c650bca-d3a9-495a-b749-30ebeef39adb None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:37.489715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c650bca-d3a9-495a-b749-30ebeef39adb None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:37.491357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 97/211] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:37.496892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c0666a-6062-4fbe-925c-1add798ad5d8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:37.497126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c0666a-6062-4fbe-925c-1add798ad5d8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:37.497401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c0666a-6062-4fbe-925c-1add798ad5d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:37.515831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b5c0666a-6062-4fbe-925c-1add798ad5d8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:37.517047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b5c0666a-6062-4fbe-925c-1add798ad5d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:37.587255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 115/212] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:37.646215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c650bca-d3a9-495a-b749-30ebeef39adb None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:37.755015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c650bca-d3a9-495a-b749-30ebeef39adb None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:37.790583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:37.791534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:37.792019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:37.792484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:37.803737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:37.809382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:37.810218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:37.811020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:37.818051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 98/213] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:37.819615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c0666a-6062-4fbe-925c-1add798ad5d8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:37.857162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:37.938503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:37.941385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:37.959623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c0666a-6062-4fbe-925c-1add798ad5d8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:37.988611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c11304-a4b9-4b04-a3a1-f057feaea940 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:38.009099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:38.010015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.010958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.011416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.012541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:38.013028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:38.015593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:38.026967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 116/214] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:38.027644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:38.028356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.028704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.029049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.030248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:38.031890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:38.032940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:38.038676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 99/215] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:38.039730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:38.151748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28feff49-2dae-4c48-877e-8e732b31bf87 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:38.172265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6a546c-0598-4900-a54d-668edc542ceb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:38.188951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:38.189681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.191170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.191440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] PATH_INFO: `/v3/users/acd7abd9a6f94a68aa6b44779329965e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.202046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:38.211934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] RBAC: Authorizing `identity:get_user(user_id=acd7abd9a6f94a68aa6b44779329965e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:38.214283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:38.216361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a43b1a-6ca0-4128-b7a4-51393179a2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:38.216846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 117/216] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:38 2022] GET /identity/v3/users/acd7abd9a6f94a68aa6b44779329965e => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:38.218190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a43b1a-6ca0-4128-b7a4-51393179a2fe None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.218919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a43b1a-6ca0-4128-b7a4-51393179a2fe None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.219600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a43b1a-6ca0-4128-b7a4-51393179a2fe None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.220848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a43b1a-6ca0-4128-b7a4-51393179a2fe None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:38.223091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a43b1a-6ca0-4128-b7a4-51393179a2fe None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:38.227804 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:38.242374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:38.355385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 100/217] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:32:38 2022] POST /identity/v3/users => generated 297 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:38.368254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbcf235-b3e7-475b-b248-d5e4fde36965 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:38.429425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:38.430497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.430980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.431491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.511192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:38.513080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:38.520468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c6feef-d990-4e20-9541-f715e33944f0 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.520793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c6feef-d990-4e20-9541-f715e33944f0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.521013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c6feef-d990-4e20-9541-f715e33944f0 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.522369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 101/218] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:38.525704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:38.527740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd461ef-9124-4aca-8fc7-a8ba356b2dc0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.528013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd461ef-9124-4aca-8fc7-a8ba356b2dc0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.528277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd461ef-9124-4aca-8fc7-a8ba356b2dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.549500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-acd461ef-9124-4aca-8fc7-a8ba356b2dc0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:38.550734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-acd461ef-9124-4aca-8fc7-a8ba356b2dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:38.642695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 118/219] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:32:38.643052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 102/220] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:38.651369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:38.779396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08396e72-7aaf-4e77-88cd-f9b5fba7e2fb None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:38.787967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bc48f-9118-4f7e-adc7-ec75ede42a5a None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.788276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bc48f-9118-4f7e-adc7-ec75ede42a5a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.788511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bc48f-9118-4f7e-adc7-ec75ede42a5a None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.790000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 103/221] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:38.791703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:38.794861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ac4fdb-47ec-4aa3-bc3c-83079e415f8e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.795143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ac4fdb-47ec-4aa3-bc3c-83079e415f8e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.795423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ac4fdb-47ec-4aa3-bc3c-83079e415f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.822634 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-06ac4fdb-47ec-4aa3-bc3c-83079e415f8e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:38.824177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-06ac4fdb-47ec-4aa3-bc3c-83079e415f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:38.829223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:38.829868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:38.830149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:38.830476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:38.831999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:38.833568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:38.835481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 119/222] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:38 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:38.843961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:38.912150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 104/223] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:38.935330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ac4fdb-47ec-4aa3-bc3c-83079e415f8e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:39.009438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:39.021517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d0d572-28b8-4869-82e2-309348c234f9 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.095938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ac4fdb-47ec-4aa3-bc3c-83079e415f8e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.100801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:39.101418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:39.101691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:39.101958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:39.103668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=acd7abd9a6f94a68aa6b44779329965e) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:39.104038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:39.105482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:39.125936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 120/224] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:38 2022] GET /identity/v3/role_assignments?user.id=acd7abd9a6f94a68aa6b44779329965e&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 259 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:39.164354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:39.164914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:39.165187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:39.165453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:39.184739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:39.190183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:39.191127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:39.191895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:39.194261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 105/225] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:39.202442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:39.222172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:39.281369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:39.353333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6eb3c1-e98d-4370-b6ac-876e8d1785c0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.413509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:39.415697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:39.417224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:39.417719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:39.418183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:39.418507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdb2664-218b-41b3-87e3-7bc5002ecaa3 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.419360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:39.420058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:39.422539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:39.438608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 121/226] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:39.442230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:39.479889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:39.480532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:39.480871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:39.481066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:39.498222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:39.503933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:39.504817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:39.505574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:39.506818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 106/227] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:32:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:39.512402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:39.654641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7c8816-434c-4368-ba22-c24afc194aa8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.660938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:39.676267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfab491-e754-4731-9b45-f7d16742052a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.717320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:39.717320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:39.717320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:39.717320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] PATH_INFO: `/v3/users/693ed507c7d3486e8bd7fc299037cd95` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:39.721703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] RBAC: Authorizing `identity:get_user(user_id=693ed507c7d3486e8bd7fc299037cd95)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:39.721703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:39.722667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 122/228] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:39 2022] GET /identity/v3/users/693ed507c7d3486e8bd7fc299037cd95 => generated 283 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:39.732784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:39.732784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:39.732784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:39.732784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:39.733140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:39.733140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:39.733256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:39.738425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:39.756285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 107/229] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:32:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:39.763443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:39.805191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:39.912664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e5f62b-dec5-46d9-ada0-3592730affc4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.948918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:39.949098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6022c9-4e1d-4045-b3ed-1a17a406e442 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:39.950064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:39.950595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:39.951064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:39.956453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:39.962707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:39.964739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 123/230] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:39 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:39.972191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:40.017179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:40.018809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:40.019309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:40.021132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] PATH_INFO: `/v3/users/693ed507c7d3486e8bd7fc299037cd95` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:40.023539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] RBAC: Authorizing `identity:get_user(user_id=693ed507c7d3486e8bd7fc299037cd95)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:40.025230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:40.030172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 108/231] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:39 2022] GET /identity/v3/users/693ed507c7d3486e8bd7fc299037cd95 => generated 283 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:40.034493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:40.050883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:40.178038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8748f6dd-3f2b-45a0-b8b8-c269921db0b2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:40.189132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:40.212839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c511d73-c82d-4c63-bb72-1ba4cd1d9fd3 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:40.231818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734d26fe-9600-46f2-890e-dcf90c0d110f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:40.232868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d26fe-9600-46f2-890e-dcf90c0d110f None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:40.233369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d26fe-9600-46f2-890e-dcf90c0d110f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:40.233885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d26fe-9600-46f2-890e-dcf90c0d110f None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b/users/693ed507c7d3486e8bd7fc299037cd95/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:40.237318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d26fe-9600-46f2-890e-dcf90c0d110f None admin] RBAC: Authorizing `identity:create_grant(project_id=1d3187c323414f38ae8026b57e57504b, user_id=693ed507c7d3486e8bd7fc299037cd95, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:40.239768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d26fe-9600-46f2-890e-dcf90c0d110f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:40.267321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:40.268896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:40.269429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:40.269899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:40.271283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:40.273021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:40.275264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 109/232] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:40 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:40.282558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:40.313743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 124/233] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:39 2022] PUT /identity/v3/projects/1d3187c323414f38ae8026b57e57504b/users/693ed507c7d3486e8bd7fc299037cd95/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:32:40.457267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c134e4-dda9-49b2-b898-d3d0655f19cd None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:40.513138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418838b7-b323-4318-8d53-90ad1a31f4b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:40.514085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418838b7-b323-4318-8d53-90ad1a31f4b7 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:40.514537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418838b7-b323-4318-8d53-90ad1a31f4b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:40.514956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418838b7-b323-4318-8d53-90ad1a31f4b7 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b/users/693ed507c7d3486e8bd7fc299037cd95/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:40.520482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418838b7-b323-4318-8d53-90ad1a31f4b7 None admin] RBAC: Authorizing `identity:create_grant(project_id=1d3187c323414f38ae8026b57e57504b, user_id=693ed507c7d3486e8bd7fc299037cd95, role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:40.522156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418838b7-b323-4318-8d53-90ad1a31f4b7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:40.630478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 110/234] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:40 2022] PUT /identity/v3/projects/1d3187c323414f38ae8026b57e57504b/users/693ed507c7d3486e8bd7fc299037cd95/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:32:41.019505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e155a-eb3b-45ec-954d-4df689f0faf3 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:41.020017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e155a-eb3b-45ec-954d-4df689f0faf3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:41.020017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e155a-eb3b-45ec-954d-4df689f0faf3 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:41.021694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 125/235] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:41.026980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa9353-d639-430a-86f6-42071b3921e0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:41.027258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa9353-d639-430a-86f6-42071b3921e0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:41.027529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa9353-d639-430a-86f6-42071b3921e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:41.059536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-bcfa9353-d639-430a-86f6-42071b3921e0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:41.060761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bcfa9353-d639-430a-86f6-42071b3921e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:41.082756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:41.156156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 111/236] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:41.184963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a84b8a-e756-4acb-927b-8d1b534d7440 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:41.185500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a84b8a-e756-4acb-927b-8d1b534d7440 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:41.185781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a84b8a-e756-4acb-927b-8d1b534d7440 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:41.187167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 126/237] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:41.193137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b0651-f0d3-4c99-9495-0912aca6874d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:41.193426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b0651-f0d3-4c99-9495-0912aca6874d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:41.193697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b0651-f0d3-4c99-9495-0912aca6874d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:41.217854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f48b0651-f0d3-4c99-9495-0912aca6874d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:41.219075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f48b0651-f0d3-4c99-9495-0912aca6874d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:41.312171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 112/238] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:41.603779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a84b8a-e756-4acb-927b-8d1b534d7440 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:41.661777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b0651-f0d3-4c99-9495-0912aca6874d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:41.772754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:41.775008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a84b8a-e756-4acb-927b-8d1b534d7440 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:41.824014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:41.828690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:41.829580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:41.830090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:41.830659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:41.841753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:41.847225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:41.848086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:41.848917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:41.851506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 127/239] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:41.864520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:41.864717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b0651-f0d3-4c99-9495-0912aca6874d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:41.926987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:41.927655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:41.927936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:41.928209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:41.931483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:41.932993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:41.940172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 113/240] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:41.941138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:42.005518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6823458-a75e-470a-8832-4f89de0ea4c8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:42.046645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb21532-8751-444d-b470-78f1a841af57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:42.048015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb21532-8751-444d-b470-78f1a841af57 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.048484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb21532-8751-444d-b470-78f1a841af57 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.048968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb21532-8751-444d-b470-78f1a841af57 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.049899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb21532-8751-444d-b470-78f1a841af57 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:42.050390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb21532-8751-444d-b470-78f1a841af57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:42.052023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb21532-8751-444d-b470-78f1a841af57 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:42.062443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 128/241] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:42.063884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30b89e9-c8b6-4d97-84bf-09fe1a776b0b None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.064359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30b89e9-c8b6-4d97-84bf-09fe1a776b0b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.065668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30b89e9-c8b6-4d97-84bf-09fe1a776b0b None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.067136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 129/242] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:42.071341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30b89e9-c8b6-4d97-84bf-09fe1a776b0b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:42.093493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfb8eb-7961-4f43-b01e-7dda88ae0a09 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:42.095456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:42.138079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccaff7a-e1e6-49b0-b670-14a66c96a307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:42.138663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccaff7a-e1e6-49b0-b670-14a66c96a307 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.139067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccaff7a-e1e6-49b0-b670-14a66c96a307 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.139347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccaff7a-e1e6-49b0-b670-14a66c96a307 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.140316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccaff7a-e1e6-49b0-b670-14a66c96a307 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:42.142016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccaff7a-e1e6-49b0-b670-14a66c96a307 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:42.181654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30b89e9-c8b6-4d97-84bf-09fe1a776b0b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:42.218103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:42.218796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.219070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.219425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 114/243] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:32:41 2022] POST /identity/v3/users => generated 293 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:42.219723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] PATH_INFO: `/v3/users/d72ba6636a0a4e6b842d15b1256ac851` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.222085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d7413-3367-4a96-b07a-3696d56831de None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.222998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d7413-3367-4a96-b07a-3696d56831de None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.223507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d7413-3367-4a96-b07a-3696d56831de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.232296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:42.249968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-503d7413-3367-4a96-b07a-3696d56831de None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:42.250880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] RBAC: Authorizing `identity:get_user(user_id=d72ba6636a0a4e6b842d15b1256ac851)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:42.251988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-503d7413-3367-4a96-b07a-3696d56831de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:42.252843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:42.255062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 130/244] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:42 2022] GET /identity/v3/users/d72ba6636a0a4e6b842d15b1256ac851 => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:42.260760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:42.350345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 115/245] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:42.417220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:42.439423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03651507-0888-4e3d-a1d1-fabf71bb5899 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:42.476389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:42.476997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.477271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.477537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.552501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:42.553973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:42.642486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:42.669548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 131/246] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:32:42.672006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503d7413-3367-4a96-b07a-3696d56831de None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:42.675439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:42.798596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf7c68c-c7a4-4af8-9d93-7ce416d9a0fb None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:42.809506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503d7413-3367-4a96-b07a-3696d56831de None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:42.848589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:42.849552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.850113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:42.850223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.850680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.852058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:42.854008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:42.856103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:42.856511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 132/247] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:42 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:42.857167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:42.857275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:42.858476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:42.863400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:42.875342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:42.882302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:42.883253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:42.884009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 116/248] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:42.891337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:42.928566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:42.991428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e1699-2a97-4cb8-a99e-b6d257be8e1e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:43.028789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49615493-1ce8-4c1d-8c7c-248faffe805a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:43.041714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:43.042822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.043378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.043920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.044919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] RBAC: Adding query filter params (role.id=35cf5ed465b441dc825c683f9e9a02a6, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=d72ba6636a0a4e6b842d15b1256ac851) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:43.045447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:43.048119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:43.056349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 133/249] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:42 2022] GET /identity/v3/role_assignments?user.id=d72ba6636a0a4e6b842d15b1256ac851&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=35cf5ed465b441dc825c683f9e9a02a6 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:43.081150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76338e16-8a1e-4f16-b0d4-c8a417db1dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:43.082147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76338e16-8a1e-4f16-b0d4-c8a417db1dca None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.082657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76338e16-8a1e-4f16-b0d4-c8a417db1dca None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.083119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76338e16-8a1e-4f16-b0d4-c8a417db1dca None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.084015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76338e16-8a1e-4f16-b0d4-c8a417db1dca None admin] RBAC: Adding query filter params (name=member) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:43.084504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76338e16-8a1e-4f16-b0d4-c8a417db1dca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:43.086118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76338e16-8a1e-4f16-b0d4-c8a417db1dca None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:43.096604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 117/250] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:43.103096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:43.183563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1b3ae2-bbf0-4994-a64a-c8ae029e90fe None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.184156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1b3ae2-bbf0-4994-a64a-c8ae029e90fe None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.184664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1b3ae2-bbf0-4994-a64a-c8ae029e90fe None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.187187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 118/251] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:43.198184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747228b9-f542-4fa9-b2f7-0f93036c7264 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.198526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747228b9-f542-4fa9-b2f7-0f93036c7264 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.198777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747228b9-f542-4fa9-b2f7-0f93036c7264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.229277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-747228b9-f542-4fa9-b2f7-0f93036c7264 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:43.230547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-747228b9-f542-4fa9-b2f7-0f93036c7264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:43.244897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dab19b-5ff9-4f16-9382-319acb5d5db7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:43.266837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:43.336661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c8b323-d2ae-4a96-b686-54287eb78139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:43.338928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c8b323-d2ae-4a96-b686-54287eb78139 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.339411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c8b323-d2ae-4a96-b686-54287eb78139 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.339961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c8b323-d2ae-4a96-b686-54287eb78139 None admin] PATH_INFO: `/v3/users/acd7abd9a6f94a68aa6b44779329965e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.343218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c8b323-d2ae-4a96-b686-54287eb78139 None admin] RBAC: Authorizing `identity:get_user(user_id=acd7abd9a6f94a68aa6b44779329965e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:43.345763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c8b323-d2ae-4a96-b686-54287eb78139 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:43.347371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:43.351869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 134/252] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:43 2022] GET /identity/v3/users/acd7abd9a6f94a68aa6b44779329965e => generated 297 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:43.351869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3e6bc5-a0f0-4a6e-9997-fd452fd631fc None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.351869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3e6bc5-a0f0-4a6e-9997-fd452fd631fc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.351869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3e6bc5-a0f0-4a6e-9997-fd452fd631fc None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.352415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 135/253] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:43.357338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3e6bc5-a0f0-4a6e-9997-fd452fd631fc None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:43.374128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 119/254] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:43.379765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d32f67-d4cb-4f6b-9d5a-6b8ea938b011 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.380047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d32f67-d4cb-4f6b-9d5a-6b8ea938b011 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.380336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d32f67-d4cb-4f6b-9d5a-6b8ea938b011 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.435640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d4d32f67-d4cb-4f6b-9d5a-6b8ea938b011 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:43.436749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d4d32f67-d4cb-4f6b-9d5a-6b8ea938b011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:43.495368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3e6bc5-a0f0-4a6e-9997-fd452fd631fc None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:43.534856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:43.535907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.536377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.536858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.538380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:43.540637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:43.542506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 136/255] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:43 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:43.550766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:43.608201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 120/256] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:43.609382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:43.669713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385a1bb-6ea8-4b0a-95a9-0120e1a09e4a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:43.706982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:43.708088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.708569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.709033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b/users/acd7abd9a6f94a68aa6b44779329965e/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.711204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] RBAC: Authorizing `identity:create_grant(project_id=1d3187c323414f38ae8026b57e57504b, user_id=acd7abd9a6f94a68aa6b44779329965e, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:43.712852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:43.789932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 137/257] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:43 2022] PUT /identity/v3/projects/1d3187c323414f38ae8026b57e57504b/users/acd7abd9a6f94a68aa6b44779329965e/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:32:43.796324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d32f67-d4cb-4f6b-9d5a-6b8ea938b011 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:43.922853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d32f67-d4cb-4f6b-9d5a-6b8ea938b011 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:43.948051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:43.975262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:43.975879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:43.976154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:43.976438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:43.988918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:43.994114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:43.995052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:43.996262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 121/258] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:32:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:44.001534 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:44.134518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfe9762-1bd6-4f7c-9c29-c29954201815 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:44.141914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:44.167111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:44.178623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:44.179266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:44.179612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:44.179897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:44.180592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:44.180899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:44.182486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:44.197519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 138/259] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:44.204041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:44.383683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52962830-5f3e-4e1d-8ebc-6cb2ab5719b8 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:44.433684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:44.434717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:44.436323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:44.437017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:44.454080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7329a3a-d81c-43ad-a619-759c06aab5bc None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:44.467264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:44.482448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:44.483397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:44.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:44.491257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 122/260] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:32:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:44.496112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:44.551080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:44.552078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:44.552728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:44.554390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] PATH_INFO: `/v3/users/693ed507c7d3486e8bd7fc299037cd95` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:44.556590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] RBAC: Authorizing `identity:get_user(user_id=693ed507c7d3486e8bd7fc299037cd95)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:44.558370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:44.563243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 139/261] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:44 2022] GET /identity/v3/users/693ed507c7d3486e8bd7fc299037cd95 => generated 283 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:44.571794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:44.583223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:44.668055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:44.726510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3b92a1-fe66-4889-8989-99af1b27c702 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:44.810980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:44.811595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:44.811856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:44.812127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:44.812798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:44.813101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:44.816406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5885a679-ccd6-48dc-adde-8727cea7c5c2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:44.819283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:44.838523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 123/262] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:32:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:44.844418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:44.894947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:44.895569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:44.895856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:44.896132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:45.022263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:45.028411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902df77b-e2a3-40ab-bd6c-7549f6a801c0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:45.086976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:45.088538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:45.091031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:45.091680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:45.091960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:45.092233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] PATH_INFO: `/v3/users/693ed507c7d3486e8bd7fc299037cd95` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:45.093414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] RBAC: Authorizing `identity:get_user(user_id=693ed507c7d3486e8bd7fc299037cd95)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:45.095722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:45.098075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 124/263] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:44 2022] GET /identity/v3/users/693ed507c7d3486e8bd7fc299037cd95 => generated 283 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:45.103921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:45.194762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:45.248485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeadd0a1-1021-43ff-bc10-ceb40b72dd85 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:45.316732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:45.317281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:45.317557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:45.317825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:45.348405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 140/264] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 766 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:32:45.355498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:45.365532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:45.428976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:45.430452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:45.534950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7818bf79-5b7a-4d57-8b5e-f1da1f6dedcc None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:45.605710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:45.606336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:45.606700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:45.606977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:45.608333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:45.609825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:45.613551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 141/265] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:45 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:45.620936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:45.629008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:45.632765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 125/266] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 530 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:32:45.639470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:45.747612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:45.813665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c81e6ee-6c7b-4407-b79f-2f699cdca76a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:45.883765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03900fd2-7b16-457b-b61b-d4a56ea11dd5 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:45.888237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57213dd8-c3ec-46d1-88fc-bc388ac9502d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:45.888906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57213dd8-c3ec-46d1-88fc-bc388ac9502d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:45.889185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57213dd8-c3ec-46d1-88fc-bc388ac9502d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:45.889457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57213dd8-c3ec-46d1-88fc-bc388ac9502d None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:45.896524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57213dd8-c3ec-46d1-88fc-bc388ac9502d None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:45.898036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57213dd8-c3ec-46d1-88fc-bc388ac9502d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:45.899936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 126/267] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:45 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:45.901192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5befee17-b1d8-4c49-baa2-8e1c00e5db5e None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:45.901473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5befee17-b1d8-4c49-baa2-8e1c00e5db5e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:45.901744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5befee17-b1d8-4c49-baa2-8e1c00e5db5e None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:45.903374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 127/268] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:45.914782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5befee17-b1d8-4c49-baa2-8e1c00e5db5e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:45.965354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b9061b-b844-45c9-8c5e-7a205438c6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:45.965984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b9061b-b844-45c9-8c5e-7a205438c6ad None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:45.966290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b9061b-b844-45c9-8c5e-7a205438c6ad None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:45.966593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b9061b-b844-45c9-8c5e-7a205438c6ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:45.967283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b9061b-b844-45c9-8c5e-7a205438c6ad None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=693ed507c7d3486e8bd7fc299037cd95) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:45.967591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b9061b-b844-45c9-8c5e-7a205438c6ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:45.969823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b9061b-b844-45c9-8c5e-7a205438c6ad None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:45.985767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 142/269] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:45 2022] GET /identity/v3/role_assignments?user.id=693ed507c7d3486e8bd7fc299037cd95&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:45.987734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085e25d-d4c1-4257-b709-dc1d4d2f5fa9 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:45.988016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085e25d-d4c1-4257-b709-dc1d4d2f5fa9 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:45.989274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085e25d-d4c1-4257-b709-dc1d4d2f5fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.022813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d085e25d-d4c1-4257-b709-dc1d4d2f5fa9 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:46.024028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d085e25d-d4c1-4257-b709-dc1d4d2f5fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:46.089983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5befee17-b1d8-4c49-baa2-8e1c00e5db5e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:46.126847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:46.136006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f358b43-ea8d-49bd-bc29-7445567ea0d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:46.136588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f358b43-ea8d-49bd-bc29-7445567ea0d2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.136867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f358b43-ea8d-49bd-bc29-7445567ea0d2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.137140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f358b43-ea8d-49bd-bc29-7445567ea0d2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.138984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f358b43-ea8d-49bd-bc29-7445567ea0d2 None admin] RBAC: Adding query filter params (role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=693ed507c7d3486e8bd7fc299037cd95) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:46.139303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f358b43-ea8d-49bd-bc29-7445567ea0d2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:46.143927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f358b43-ea8d-49bd-bc29-7445567ea0d2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:46.164740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 128/270] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:45 2022] GET /identity/v3/role_assignments?user.id=693ed507c7d3486e8bd7fc299037cd95&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 608 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:46.166031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b8f42-6da3-466d-83ab-fa4e0707e0a1 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.166309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b8f42-6da3-466d-83ab-fa4e0707e0a1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.166600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1b8f42-6da3-466d-83ab-fa4e0707e0a1 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.168049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 129/271] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:46.174164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc022b7e-2a7d-42fa-b56b-31142ba0fc82 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.174437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc022b7e-2a7d-42fa-b56b-31142ba0fc82 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.175177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc022b7e-2a7d-42fa-b56b-31142ba0fc82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.184886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:46.191252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 143/272] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:46.214912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-cc022b7e-2a7d-42fa-b56b-31142ba0fc82 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:46.216136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cc022b7e-2a7d-42fa-b56b-31142ba0fc82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:46.316467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 130/273] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:46.435190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6d2d6-1e8c-47ff-8776-0397d432fed5 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.435706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6d2d6-1e8c-47ff-8776-0397d432fed5 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.435809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd6d2d6-1e8c-47ff-8776-0397d432fed5 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.437118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 144/274] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:46.443397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6de4a5f-79ab-4bda-bbeb-ac76e4cb542e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.443685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6de4a5f-79ab-4bda-bbeb-ac76e4cb542e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.443957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6de4a5f-79ab-4bda-bbeb-ac76e4cb542e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.472142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b6de4a5f-79ab-4bda-bbeb-ac76e4cb542e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:46.473334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b6de4a5f-79ab-4bda-bbeb-ac76e4cb542e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:46.511049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd6d2d6-1e8c-47ff-8776-0397d432fed5 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:46.556152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 131/275] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:46.654126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd6d2d6-1e8c-47ff-8776-0397d432fed5 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:46.656954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6de4a5f-79ab-4bda-bbeb-ac76e4cb542e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:46.720040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:46.721726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.722578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.723238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.724762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:46.727343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:46.729463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 145/276] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:46.738779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:46.766790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:46.807540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6de4a5f-79ab-4bda-bbeb-ac76e4cb542e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:46.823134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:46.849390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:46.850867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.851386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.851945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.863553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:46.868720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:46.869558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 132/277] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:46.872017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:46.890658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae232ac-3579-4783-a024-05f2bf8b241d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:46.951697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:46.952974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:46.953790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:46.954599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:46.956768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:46.959962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:46.972633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad4c776-a289-4532-8034-447bcc4ce377 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:46.988537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:47.005810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.006458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.006733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.006999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.018830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:47.024234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:47.025289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 133/278] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:47.027206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.061248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 146/279] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:32:46 2022] POST /identity/v3/users => generated 293 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:47.062114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.162698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdbaab1-ca4b-4e4a-bc92-b3c460a4de33 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:47.184086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272e5123-2545-4ed1-aa7c-47e314fc38fe None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:47.201816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e43a6ae-1523-4dab-97b0-c7138328a206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.202512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e43a6ae-1523-4dab-97b0-c7138328a206 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.202795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e43a6ae-1523-4dab-97b0-c7138328a206 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.203066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e43a6ae-1523-4dab-97b0-c7138328a206 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.203766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e43a6ae-1523-4dab-97b0-c7138328a206 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:47.204069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e43a6ae-1523-4dab-97b0-c7138328a206 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:47.205541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e43a6ae-1523-4dab-97b0-c7138328a206 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:47.212280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:47.220229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 134/280] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:47.220229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfb27a-aaa4-40f0-a2fe-d31e4510e498 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.220229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfb27a-aaa4-40f0-a2fe-d31e4510e498 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.220229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfb27a-aaa4-40f0-a2fe-d31e4510e498 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.220229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 135/281] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:47.223386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cfb27a-aaa4-40f0-a2fe-d31e4510e498 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.232195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4496954-749a-4c0c-8624-8a4948063e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.232934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4496954-749a-4c0c-8624-8a4948063e96 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.233097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4496954-749a-4c0c-8624-8a4948063e96 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.233335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4496954-749a-4c0c-8624-8a4948063e96 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.234355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4496954-749a-4c0c-8624-8a4948063e96 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:47.234355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4496954-749a-4c0c-8624-8a4948063e96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:47.235745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4496954-749a-4c0c-8624-8a4948063e96 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:47.246462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 147/282] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:47.247713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406072d5-3781-4c6d-a191-d81a7f3482f4 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.248022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406072d5-3781-4c6d-a191-d81a7f3482f4 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.248263 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406072d5-3781-4c6d-a191-d81a7f3482f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.268394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-406072d5-3781-4c6d-a191-d81a7f3482f4 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:47.269827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-406072d5-3781-4c6d-a191-d81a7f3482f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:47.275499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:47.349935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 148/283] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:47.350620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406072d5-3781-4c6d-a191-d81a7f3482f4 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.356256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cfb27a-aaa4-40f0-a2fe-d31e4510e498 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:47.410186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.411164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.411452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.411725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] PATH_INFO: `/v3/users/d72ba6636a0a4e6b842d15b1256ac851` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.413290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] RBAC: Authorizing `identity:get_user(user_id=d72ba6636a0a4e6b842d15b1256ac851)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:47.415263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:47.417551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 136/284] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:47 2022] GET /identity/v3/users/d72ba6636a0a4e6b842d15b1256ac851 => generated 297 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:47.423089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.496207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406072d5-3781-4c6d-a191-d81a7f3482f4 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:47.515182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:47.543120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.544035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.544318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.544592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] PATH_INFO: `/v3/users/18008430963348e69383c358dbc1fd34` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.545730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8377c7-18f9-4c1e-8a42-1c5e0cfe2ef2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:47.573607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] RBAC: Authorizing `identity:get_user(user_id=18008430963348e69383c358dbc1fd34)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:47.574749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:47.575119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:47.576983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 149/285] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:47 2022] GET /identity/v3/users/18008430963348e69383c358dbc1fd34 => generated 293 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:47.587232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.622487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.623195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.623482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.623748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.624918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:47.626365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:47.628016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 137/286] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:47 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:47.633726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.727019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b44cfd-d515-4f1c-9c58-7739f0c46644 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:47.767389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef1fbd8-bc68-4934-95f2-0e3b8ecffa10 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:47.771117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.771729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.772004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.772272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.772993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, user.id=18008430963348e69383c358dbc1fd34, scope.system=all) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:47.773296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:47.774774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:47.780354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 150/287] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Oct 4 08:32:47 2022] GET /identity/v3/role_assignments?user.id=18008430963348e69383c358dbc1fd34&scope.system=all&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:47.781169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:47.819993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:47.820688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:47.820962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:47.821228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b/users/d72ba6636a0a4e6b842d15b1256ac851/roles/35cf5ed465b441dc825c683f9e9a02a6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:47.823318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] RBAC: Authorizing `identity:create_grant(project_id=1d3187c323414f38ae8026b57e57504b, user_id=d72ba6636a0a4e6b842d15b1256ac851, role_id=35cf5ed465b441dc825c683f9e9a02a6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:47.824786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:47.837961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:47.912920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 138/288] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:32:47 2022] PUT /identity/v3/projects/1d3187c323414f38ae8026b57e57504b/users/d72ba6636a0a4e6b842d15b1256ac851/roles/35cf5ed465b441dc825c683f9e9a02a6 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:47.935404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4df8539-828a-4993-8b51-93d21e4bb7c4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:48.024962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:48.025538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:48.025820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:48.026086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:48.051534 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:48.056584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:48.057445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:48.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 151/289] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:48.065665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:48.130809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:48.243551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7767e140-37d6-4cd5-aa74-28d0d8254d77 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:48.327227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:48.327779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:48.328113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:48.328389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:48.329070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] RBAC: Adding query filter params (name=member) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:48.329370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:48.331656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:48.347919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 139/290] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:48.357462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:48.531393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:48.537639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d44673d-e024-4c6d-a4ac-99de0adf71c0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:48.581463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:48.582015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:48.582291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:48.582590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] PATH_INFO: `/v3/users/acd7abd9a6f94a68aa6b44779329965e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:48.583736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] RBAC: Authorizing `identity:get_user(user_id=acd7abd9a6f94a68aa6b44779329965e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:48.585235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:48.587099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 152/291] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:48 2022] GET /identity/v3/users/acd7abd9a6f94a68aa6b44779329965e => generated 297 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:48.592658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:48.696917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:48.703092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631666b8-4dde-4640-8f3b-4f330c697ad8 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:48.738208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:48.738847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:48.739122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:48.739418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:48.836602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:48.837992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:48.961153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:48.967803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 140/292] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:48.974001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:49.101960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002301a1-6ca4-4096-8a67-e98dfa159a87 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:49.137522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aa2433-f80e-4d26-bc79-14f701104fe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:49.138152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa2433-f80e-4d26-bc79-14f701104fe9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:49.138447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa2433-f80e-4d26-bc79-14f701104fe9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:49.138721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa2433-f80e-4d26-bc79-14f701104fe9 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:49.139845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aa2433-f80e-4d26-bc79-14f701104fe9 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:49.141287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aa2433-f80e-4d26-bc79-14f701104fe9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:49.143941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 153/293] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:48 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:49.149855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:49.281798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2bea67-2c2e-4937-ac3a-7eb04ed40a57 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:49.329419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e00cfe4-94ff-4978-9b02-c54f582611f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:49.330023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00cfe4-94ff-4978-9b02-c54f582611f0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:49.330991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00cfe4-94ff-4978-9b02-c54f582611f0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:49.331697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00cfe4-94ff-4978-9b02-c54f582611f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:49.332431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e00cfe4-94ff-4978-9b02-c54f582611f0 None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=acd7abd9a6f94a68aa6b44779329965e) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:49.332737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e00cfe4-94ff-4978-9b02-c54f582611f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:49.334237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e00cfe4-94ff-4978-9b02-c54f582611f0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:49.340526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 141/294] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:49 2022] GET /identity/v3/role_assignments?user.id=acd7abd9a6f94a68aa6b44779329965e&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:49.969143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11304a9d-fc54-4c6b-bb35-6f336c4c218a None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:49.969669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11304a9d-fc54-4c6b-bb35-6f336c4c218a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:49.969669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11304a9d-fc54-4c6b-bb35-6f336c4c218a None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:49.971127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 154/295] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:49.976505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f04a15-a9b1-4053-b1fe-de0868981766 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:49.976902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f04a15-a9b1-4053-b1fe-de0868981766 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:49.977080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f04a15-a9b1-4053-b1fe-de0868981766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:49.996905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e5f04a15-a9b1-4053-b1fe-de0868981766 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:49.998106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e5f04a15-a9b1-4053-b1fe-de0868981766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:50.039595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc263d00-62e2-4acd-a2a6-e15b99ea6d0b None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.040201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc263d00-62e2-4acd-a2a6-e15b99ea6d0b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.040696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc263d00-62e2-4acd-a2a6-e15b99ea6d0b None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.042541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 155/296] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:50.048206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772f852-cf6a-4228-8ee4-10b681a37f83 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.048769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772f852-cf6a-4228-8ee4-10b681a37f83 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.049774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772f852-cf6a-4228-8ee4-10b681a37f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.062698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:50.071146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1772f852-cf6a-4228-8ee4-10b681a37f83 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:50.072705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1772f852-cf6a-4228-8ee4-10b681a37f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:50.077039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 142/297] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:50.112661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:50.161424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 156/298] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:50.355628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f04a15-a9b1-4053-b1fe-de0868981766 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:50.444378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772f852-cf6a-4228-8ee4-10b681a37f83 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:50.465849 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f04a15-a9b1-4053-b1fe-de0868981766 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:50.509480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:50.510549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.511029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.511594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.513015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:50.514820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:50.517002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 143/299] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:50.523255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:50.583922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772f852-cf6a-4228-8ee4-10b681a37f83 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:50.608829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:50.633163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:50.634548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.634828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.635101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.650162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:50.657387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:50.658935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:50.660376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 157/300] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:50.665206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:50.666036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e988ed-95cc-4961-888d-445fd3dfd359 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:50.692075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:50.704925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e646955-80b4-4779-bbed-adb3b850ba59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:50.706543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e646955-80b4-4779-bbed-adb3b850ba59 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.706832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e646955-80b4-4779-bbed-adb3b850ba59 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.707111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e646955-80b4-4779-bbed-adb3b850ba59 None admin] PATH_INFO: `/v3/groups` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.707751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e646955-80b4-4779-bbed-adb3b850ba59 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:50.709217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e646955-80b4-4779-bbed-adb3b850ba59 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:50.766392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68265534-081b-49b9-8424-037e118a6ef7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:50.794847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 144/301] 10.222.0.27 () {60 vars in 1134 bytes} [Tue Oct 4 08:32:50 2022] POST /identity/v3/groups => generated 229 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:32:50.796100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca810384-3818-42fe-bd40-42358e57a33b None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.796401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca810384-3818-42fe-bd40-42358e57a33b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.796686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca810384-3818-42fe-bd40-42358e57a33b None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.797955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 145/302] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:50.799084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fe03f3-9cae-4acf-a408-4c2347d6c4b3 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.799356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fe03f3-9cae-4acf-a408-4c2347d6c4b3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.799627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fe03f3-9cae-4acf-a408-4c2347d6c4b3 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.805013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 146/303] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:50.805013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7905f23-d345-4170-bdeb-1920a9d01623 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.805283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7905f23-d345-4170-bdeb-1920a9d01623 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.805539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7905f23-d345-4170-bdeb-1920a9d01623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.812363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda1c69-8608-45c7-904d-4cd8f7435ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:50.814032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda1c69-8608-45c7-904d-4cd8f7435ef9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.814246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda1c69-8608-45c7-904d-4cd8f7435ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.814624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda1c69-8608-45c7-904d-4cd8f7435ef9 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.815332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda1c69-8608-45c7-904d-4cd8f7435ef9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:50.815631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda1c69-8608-45c7-904d-4cd8f7435ef9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:50.819669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda1c69-8608-45c7-904d-4cd8f7435ef9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:50.824833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b7905f23-d345-4170-bdeb-1920a9d01623 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:50.826252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b7905f23-d345-4170-bdeb-1920a9d01623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:50.827436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 158/304] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:50.828742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc50a6d-2f04-44bc-b86f-ed237f20b2a8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:50.829026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc50a6d-2f04-44bc-b86f-ed237f20b2a8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:50.829395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc50a6d-2f04-44bc-b86f-ed237f20b2a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:50.849847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0cc50a6d-2f04-44bc-b86f-ed237f20b2a8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:50.851142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0cc50a6d-2f04-44bc-b86f-ed237f20b2a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:50.906828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 147/305] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:50.908046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7905f23-d345-4170-bdeb-1920a9d01623 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:50.942696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 159/306] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:50.981598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.042575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7905f23-d345-4170-bdeb-1920a9d01623 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.078248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.078924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.079201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.079538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] PATH_INFO: `/v3/users/55dbcbf5935d48c6b17c6133159f57c7` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.098902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] RBAC: Authorizing `identity:get_user(user_id=55dbcbf5935d48c6b17c6133159f57c7)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:51.100338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:51.102048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 148/307] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:50 2022] GET /identity/v3/users/55dbcbf5935d48c6b17c6133159f57c7 => generated 293 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:51.107853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc50a6d-2f04-44bc-b86f-ed237f20b2a8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.123797 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.214041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.214719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc50a6d-2f04-44bc-b86f-ed237f20b2a8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.249712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.250316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.250616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.250884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.251604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] RBAC: Adding query filter params (role.id=35cf5ed465b441dc825c683f9e9a02a6, user.id=55dbcbf5935d48c6b17c6133159f57c7, scope.system=all) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:51.251912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:51.254057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:51.259131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 160/308] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/role_assignments?user.id=55dbcbf5935d48c6b17c6133159f57c7&scope.system=all&role.id=35cf5ed465b441dc825c683f9e9a02a6 => generated 226 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:51.259131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.336071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbf0d8f-3511-4e3c-8b01-3dc7a1babe4c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.345122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.380592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.382351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.383230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.383848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.384259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.399707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:51.405781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:51.407000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:51.407962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:51.409285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7d7bc1-a8fc-466d-a6c8-c34c7f98ec8a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.410645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 149/309] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:51.419060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.460569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.461492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.461915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.462326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:51.487548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:51.488594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:51.489545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:51.497314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 161/310] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:51.498029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.540005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb43289-fad1-4162-ac9b-d41dfdeeb514 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.579376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.579983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.580952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.581228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.581968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] RBAC: Adding query filter params (name=member) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:51.582268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:51.583745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:51.594169 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 150/311] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:51.600184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.631390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.634671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67533f13-1f89-462a-9a54-7ed023fb9050 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.674816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.675270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.675551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.675815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.676476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] RBAC: Adding query filter params (name=reader) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:51.676780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:51.678219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:51.684374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.688197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 162/312] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:51.694455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.721714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95889dba-b8b0-4f2c-b502-d754d25afd53 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.766128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.767348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.768426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.769113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] PATH_INFO: `/v3/users/18008430963348e69383c358dbc1fd34` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.770835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] RBAC: Authorizing `identity:get_user(user_id=18008430963348e69383c358dbc1fd34)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:51.773152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:51.775650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 151/313] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/users/18008430963348e69383c358dbc1fd34 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:51.783977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.824147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62af7361-7aab-4099-a651-eae266216e8b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.869137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.870132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.870791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.871065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] PATH_INFO: `/v3/users/d72ba6636a0a4e6b842d15b1256ac851` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.873825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] RBAC: Authorizing `identity:get_user(user_id=d72ba6636a0a4e6b842d15b1256ac851)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:51.875347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:51.881926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 163/314] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/users/d72ba6636a0a4e6b842d15b1256ac851 => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:51.885009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:51.905999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.913238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1f75bd-ead1-4ab6-af70-bae5ed8ea79d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:51.952824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:51.954180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:51.954908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:51.955990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] PATH_INFO: `/v3/system/users/18008430963348e69383c358dbc1fd34/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:51.957268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:51.957708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=18008430963348e69383c358dbc1fd34, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:51.960331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:51.990370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33644cc5-9e03-4197-b87d-f206c403159e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:52.027065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:52.027654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:52.027927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:52.028205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:52.028894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 152/315] 10.222.0.27 () {58 vars in 1334 bytes} [Tue Oct 4 08:32:51 2022] PUT /identity/v3/system/users/18008430963348e69383c358dbc1fd34/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:52.113293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:52.114856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:52.197125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:52.237939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 164/316] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:32:52.242066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:52.357694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31921846-2d7c-4fc7-861e-0ec696d6c087 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:52.393954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29405d2-a7aa-4497-a368-ef55521cc503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:52.394620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29405d2-a7aa-4497-a368-ef55521cc503 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:52.394900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29405d2-a7aa-4497-a368-ef55521cc503 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:52.395173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29405d2-a7aa-4497-a368-ef55521cc503 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:52.396572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b29405d2-a7aa-4497-a368-ef55521cc503 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:52.397738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b29405d2-a7aa-4497-a368-ef55521cc503 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:52.399716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 153/317] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:52 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:52.405295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:52.513587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dae7be-9942-4584-9c05-38236b5c12ab None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:52.552950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b8823d-dd44-4592-ac49-ec88c8b3a60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:52.553927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b8823d-dd44-4592-ac49-ec88c8b3a60b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:52.554447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b8823d-dd44-4592-ac49-ec88c8b3a60b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:52.554916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b8823d-dd44-4592-ac49-ec88c8b3a60b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:52.555857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b8823d-dd44-4592-ac49-ec88c8b3a60b None admin] RBAC: Adding query filter params (role.id=35cf5ed465b441dc825c683f9e9a02a6, scope.project.id=1d3187c323414f38ae8026b57e57504b, user.id=d72ba6636a0a4e6b842d15b1256ac851) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:52.556359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b8823d-dd44-4592-ac49-ec88c8b3a60b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:52.559273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b8823d-dd44-4592-ac49-ec88c8b3a60b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:52.571250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 165/318] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:32:52 2022] GET /identity/v3/role_assignments?user.id=d72ba6636a0a4e6b842d15b1256ac851&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=35cf5ed465b441dc825c683f9e9a02a6 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:53.104265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198330b4-1f21-4c65-bb3d-177104aca17f None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:53.104781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198330b4-1f21-4c65-bb3d-177104aca17f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:53.104781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198330b4-1f21-4c65-bb3d-177104aca17f None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:53.106263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 154/319] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:53.112031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef32ae4-2477-4f21-91fe-fa1d8fafea4f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:53.112314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef32ae4-2477-4f21-91fe-fa1d8fafea4f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:53.112578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef32ae4-2477-4f21-91fe-fa1d8fafea4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:53.131800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6ef32ae4-2477-4f21-91fe-fa1d8fafea4f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:53.132976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6ef32ae4-2477-4f21-91fe-fa1d8fafea4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:53.158712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:53.206261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 166/320] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:53.473495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198330b4-1f21-4c65-bb3d-177104aca17f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:53.510699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:53.588544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198330b4-1f21-4c65-bb3d-177104aca17f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:53.616953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac9c55d-888a-4efc-8e14-c27245c2dc96 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:53.617256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac9c55d-888a-4efc-8e14-c27245c2dc96 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:53.617499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac9c55d-888a-4efc-8e14-c27245c2dc96 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:53.619261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 167/321] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:53.623609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:53.624013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3e14b3-5a88-4a17-81f9-d97de142c20b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:53.624233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:53.624331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3e14b3-5a88-4a17-81f9-d97de142c20b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:53.624500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:53.624594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3e14b3-5a88-4a17-81f9-d97de142c20b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:53.624768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:53.625887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:53.627372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:53.629031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 155/322] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:32:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:53.634630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:53.678580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ce3e14b3-5a88-4a17-81f9-d97de142c20b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:53.679682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ce3e14b3-5a88-4a17-81f9-d97de142c20b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:53.734686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:53.741423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d7f5e5-da82-4f60-b072-b618c038210e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:53.748486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 168/323] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:53.776977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:53.777582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:53.777854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:53.778120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] PATH_INFO: `/v3/groups` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:53.778767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:53.780175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:53.886614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 156/324] 10.222.0.27 () {60 vars in 1134 bytes} [Tue Oct 4 08:32:53 2022] POST /identity/v3/groups => generated 226 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:32:54.025269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3e14b3-5a88-4a17-81f9-d97de142c20b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:54.122870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:54.132087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3e14b3-5a88-4a17-81f9-d97de142c20b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:54.170273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e3131-be9f-4000-9e8f-fc1fcae2484b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:54.170273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e3131-be9f-4000-9e8f-fc1fcae2484b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:54.170273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e3131-be9f-4000-9e8f-fc1fcae2484b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:54.170273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e3131-be9f-4000-9e8f-fc1fcae2484b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:54.183775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e3131-be9f-4000-9e8f-fc1fcae2484b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-b59e3131-be9f-4000-9e8f-fc1fcae2484b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:54.189150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:54.195552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:54.196366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 169/325] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:54.198561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:54.301301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51f69b8-e2d7-4f03-83c1-5792871cc9df None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:54.301496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51f69b8-e2d7-4f03-83c1-5792871cc9df None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:54.301776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51f69b8-e2d7-4f03-83c1-5792871cc9df None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:54.303402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 170/326] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:54.308667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7042a7ba-35c7-42d7-bfcf-65011c1b403f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:54.308822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011631af-b2fa-4322-98da-803774379530 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:54.309056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011631af-b2fa-4322-98da-803774379530 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:54.309336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011631af-b2fa-4322-98da-803774379530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:54.332764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-011631af-b2fa-4322-98da-803774379530 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:54.335189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-011631af-b2fa-4322-98da-803774379530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:54.349499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:54.350118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:54.350394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:54.351117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:54.352072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] RBAC: Adding query filter params (name=reader) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:54.352376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:54.354023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:54.366352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 157/327] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:54.372458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:54.416028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:54.431307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 171/328] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:54.519050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49398bf-c6d7-4f74-8735-6b9673c6513c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:54.562284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:54.562894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:54.563168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:54.563435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] PATH_INFO: `/v3/users/55dbcbf5935d48c6b17c6133159f57c7` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:54.564896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] RBAC: Authorizing `identity:get_user(user_id=55dbcbf5935d48c6b17c6133159f57c7)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:54.567148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:54.568473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 158/329] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:54 2022] GET /identity/v3/users/55dbcbf5935d48c6b17c6133159f57c7 => generated 293 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:54.574026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011631af-b2fa-4322-98da-803774379530 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:54.718348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:54.733985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:54.739254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011631af-b2fa-4322-98da-803774379530 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:54.825641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:54.825641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:54.825641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:54.825641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] PATH_INFO: `/v3/system/users/55dbcbf5935d48c6b17c6133159f57c7/roles/35cf5ed465b441dc825c683f9e9a02a6` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:54.827098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=55dbcbf5935d48c6b17c6133159f57c7, role_id=35cf5ed465b441dc825c683f9e9a02a6)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:54.828486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:54.874011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:54.889659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a671d6e-39fb-40fd-88b2-08214d5da83f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:54.913078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 172/330] 10.222.0.27 () {58 vars in 1334 bytes} [Tue Oct 4 08:32:54 2022] PUT /identity/v3/system/users/55dbcbf5935d48c6b17c6133159f57c7/roles/35cf5ed465b441dc825c683f9e9a02a6 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:32:54.936249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:54.936915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:54.937195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:54.937468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:54.953441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:54.961108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:54.966742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:54.967512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 159/331] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:54.969553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:55.137178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b755bf-527c-474c-b38e-0a0fe083c34d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:55.181049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:55.181664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:55.181942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:55.182210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:55.182924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:55.183222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:55.184647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:55.194450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 173/332] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:55.200193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:55.334014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d96676-3dcf-4a75-a534-21100f9eb897 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:55.380125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ee95db-e37f-4fd8-bd9e-bc888b707e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:55.380771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ee95db-e37f-4fd8-bd9e-bc888b707e25 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:55.381051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ee95db-e37f-4fd8-bd9e-bc888b707e25 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:55.381322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ee95db-e37f-4fd8-bd9e-bc888b707e25 None admin] PATH_INFO: `/v3/users/18008430963348e69383c358dbc1fd34` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:55.387138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ee95db-e37f-4fd8-bd9e-bc888b707e25 None admin] RBAC: Authorizing `identity:get_user(user_id=18008430963348e69383c358dbc1fd34)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:55.388667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ee95db-e37f-4fd8-bd9e-bc888b707e25 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:55.390263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 160/333] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:32:55 2022] GET /identity/v3/users/18008430963348e69383c358dbc1fd34 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:55.399404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:55.423689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:55.543959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77832eab-373f-47bf-848e-c7cd21df68d3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:55.584994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70522bad-69cc-4325-986f-149cac12ed7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:55.585675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70522bad-69cc-4325-986f-149cac12ed7e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:55.591788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70522bad-69cc-4325-986f-149cac12ed7e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:55.591788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70522bad-69cc-4325-986f-149cac12ed7e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:55.591788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70522bad-69cc-4325-986f-149cac12ed7e None admin] RBAC: Adding query filter params (role.id=40b933f8a46e4be9bd70366ec14885e0, user.id=18008430963348e69383c358dbc1fd34, scope.system=all) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:55.591788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70522bad-69cc-4325-986f-149cac12ed7e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:55.591788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70522bad-69cc-4325-986f-149cac12ed7e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:55.606069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 174/334] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Oct 4 08:32:55 2022] GET /identity/v3/role_assignments?user.id=18008430963348e69383c358dbc1fd34&scope.system=all&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 510 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:56.668004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d0c15-c88b-4605-baea-ab80332d9f4d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:56.668507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d0c15-c88b-4605-baea-ab80332d9f4d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:56.668507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d0c15-c88b-4605-baea-ab80332d9f4d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:56.669921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 161/335] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:56.675995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7a272e-9c2f-4e0d-976b-6feeee9f067e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:56.676933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7a272e-9c2f-4e0d-976b-6feeee9f067e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:56.677436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7a272e-9c2f-4e0d-976b-6feeee9f067e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:56.697366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-dc7a272e-9c2f-4e0d-976b-6feeee9f067e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:56.699025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-dc7a272e-9c2f-4e0d-976b-6feeee9f067e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:56.775838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:56.785334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 175/336] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:57.122511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48d0c15-c88b-4605-baea-ab80332d9f4d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:57.148085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:57.252417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48d0c15-c88b-4605-baea-ab80332d9f4d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:57.296085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:57.296722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:57.297000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:57.297274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:57.311602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:57.317121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:57.317977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:57.318771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 162/337] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:32:57.324506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7a272e-9c2f-4e0d-976b-6feeee9f067e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:57.460337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7a272e-9c2f-4e0d-976b-6feeee9f067e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:57.500673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffd992f-7329-4d55-9523-89ad42ea9106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:57.501293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd992f-7329-4d55-9523-89ad42ea9106 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:57.501574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd992f-7329-4d55-9523-89ad42ea9106 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:57.502664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd992f-7329-4d55-9523-89ad42ea9106 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:57.503106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffd992f-7329-4d55-9523-89ad42ea9106 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:57.503410 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffd992f-7329-4d55-9523-89ad42ea9106 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:57.504785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffd992f-7329-4d55-9523-89ad42ea9106 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:57.515384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 176/338] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:57.521188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:57.607149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:57.635197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db11a1b9-b631-4d26-a8ea-253ffb6d6d7d None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:57.635693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8401f0c0-7f12-43aa-9fcb-db955828664f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:57.636051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db11a1b9-b631-4d26-a8ea-253ffb6d6d7d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:57.636554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db11a1b9-b631-4d26-a8ea-253ffb6d6d7d None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:57.638394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 177/339] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:57.643714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18262af6-bd75-4ccb-b2b5-d69a6ad2c2c0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:57.644250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18262af6-bd75-4ccb-b2b5-d69a6ad2c2c0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:57.644728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18262af6-bd75-4ccb-b2b5-d69a6ad2c2c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:57.678741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-18262af6-bd75-4ccb-b2b5-d69a6ad2c2c0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:57.680919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-18262af6-bd75-4ccb-b2b5-d69a6ad2c2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:57.693059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:57.693680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:57.694118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:57.694570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:57.782068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:57.805503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 178/340] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:57.846560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:57.848131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:57.909078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd1e813-d532-49b0-b493-3e32d8e5f8ee None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:57.909686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd1e813-d532-49b0-b493-3e32d8e5f8ee None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:57.910162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd1e813-d532-49b0-b493-3e32d8e5f8ee None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:57.913522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 179/341] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:32:57.918694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bab43-82af-4818-9360-d8efd259c45f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:57.919208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bab43-82af-4818-9360-d8efd259c45f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:57.919694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885bab43-82af-4818-9360-d8efd259c45f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:57.950968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:57.952374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-885bab43-82af-4818-9360-d8efd259c45f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:57.953580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-885bab43-82af-4818-9360-d8efd259c45f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:58.003092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 163/342] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:58.008700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:58.090182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 180/343] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:32:58.166553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fd6ed3-35ff-4cd4-ac44-82f55a8f21ea None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:58.218171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:58.218836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:58.219109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:58.219372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:58.221005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:58.222468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:58.225105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 164/344] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:58.229604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bab43-82af-4818-9360-d8efd259c45f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:58.356663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885bab43-82af-4818-9360-d8efd259c45f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:58.362646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:58.382443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:58.416301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:58.417171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:58.417502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:58.417771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:58.429526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:58.431147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:58.434045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 181/345] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:58.438815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:58.444118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:58.523057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691a664-4cf8-4f26-961b-6851abbb98f7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:58.568625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:58.569548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:58.569982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:58.570396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:58.591960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:58.602494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:58.603578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:32:58.604532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:32:58.605190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c634343d-87a5-43e5-814f-595560ce17f0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:58.606823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 165/346] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:58.613484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:58.652716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:58.653340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:58.653615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:58.653882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:58.654705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=7536b89e67c444c584e6eb0afdb7c343) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:58.655015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:58.656704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:58.668516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 182/347] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 260 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:58.703264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:58.793497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:58.850773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545826b-88b4-4a44-9bf0-458775cad539 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:58.856554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:58.898246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a4276-82c8-4a16-b653-090003870c1d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:58.927957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:58.928799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:58.929215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:58.929633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:58.930480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:58.930916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:58.932412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:58.945696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 166/348] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:58.953144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:58.957352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:58.958123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:58.958420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:58.958695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] PATH_INFO: `/v3/roles/member` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:58.977124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:32:58.982975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:32:58.983904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:32:58.984718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:32:58.987940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 183/349] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:32:58.990083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:59.085608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a406023-1013-4abe-8316-8e45f57ddb34 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:59.138915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57695b-e80b-48f3-8a70-95a6691f992a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:59.139767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad05d41-f207-4d4e-b6b6-ffc3affaabda None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:59.139945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57695b-e80b-48f3-8a70-95a6691f992a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.140365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57695b-e80b-48f3-8a70-95a6691f992a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.140839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57695b-e80b-48f3-8a70-95a6691f992a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.149378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:59.184591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4bac1a-a424-4abf-9be3-aadd21a7cab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:59.185123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4bac1a-a424-4abf-9be3-aadd21a7cab5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.185784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4bac1a-a424-4abf-9be3-aadd21a7cab5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.186074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4bac1a-a424-4abf-9be3-aadd21a7cab5 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.187077 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4bac1a-a424-4abf-9be3-aadd21a7cab5 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:32:59.187397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4bac1a-a424-4abf-9be3-aadd21a7cab5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:59.188962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4bac1a-a424-4abf-9be3-aadd21a7cab5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:59.190055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:59.203022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 184/350] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:32:59.204383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6a34ab-588e-4c0a-ae40-7dcae5924ed2 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.204675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6a34ab-588e-4c0a-ae40-7dcae5924ed2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.205011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6a34ab-588e-4c0a-ae40-7dcae5924ed2 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.210031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 185/351] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:59.210031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2622869-f525-4c49-96f3-1294f538e52e None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.210031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2622869-f525-4c49-96f3-1294f538e52e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.210031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2622869-f525-4c49-96f3-1294f538e52e None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.211251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 186/352] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:59.212416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2622869-f525-4c49-96f3-1294f538e52e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:59.249101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57695b-e80b-48f3-8a70-95a6691f992a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:59.251326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57695b-e80b-48f3-8a70-95a6691f992a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:59.352460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2622869-f525-4c49-96f3-1294f538e52e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:59.373004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 167/353] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:59.374373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d932bea-db94-4f43-a813-b531ba6699dd None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.374680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d932bea-db94-4f43-a813-b531ba6699dd None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.374954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d932bea-db94-4f43-a813-b531ba6699dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.396631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1d932bea-db94-4f43-a813-b531ba6699dd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:59.398025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1d932bea-db94-4f43-a813-b531ba6699dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:59.403860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322f46bf-fad5-4cdd-b0e3-c73302282ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:59.404484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322f46bf-fad5-4cdd-b0e3-c73302282ab0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.404858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322f46bf-fad5-4cdd-b0e3-c73302282ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.405139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322f46bf-fad5-4cdd-b0e3-c73302282ab0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.448396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:59.470138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:59.480780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 168/354] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:59.482275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee329d-7795-43c2-9858-c4dbe5c63d71 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.482590 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee329d-7795-43c2-9858-c4dbe5c63d71 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.482928 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee329d-7795-43c2-9858-c4dbe5c63d71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.496433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322f46bf-fad5-4cdd-b0e3-c73302282ab0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:59.498459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322f46bf-fad5-4cdd-b0e3-c73302282ab0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:59.507577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-acee329d-7795-43c2-9858-c4dbe5c63d71 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:59.508898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-acee329d-7795-43c2-9858-c4dbe5c63d71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:59.594466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 169/355] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:59.595499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee329d-7795-43c2-9858-c4dbe5c63d71 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:59.672879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 187/356] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:32:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:32:59.675117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e7ce9-bbea-4b6c-ac4f-e05a32393900 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.675584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e7ce9-bbea-4b6c-ac4f-e05a32393900 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.676021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e7ce9-bbea-4b6c-ac4f-e05a32393900 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.677421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 188/357] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:32:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:32:59.683144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cb931b-0d90-43b4-a41c-ef1acef8884e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.683428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cb931b-0d90-43b4-a41c-ef1acef8884e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.683682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cb931b-0d90-43b4-a41c-ef1acef8884e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.719509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b1cb931b-0d90-43b4-a41c-ef1acef8884e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:32:59.720813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b1cb931b-0d90-43b4-a41c-ef1acef8884e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:32:59.760053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee329d-7795-43c2-9858-c4dbe5c63d71 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:32:59.813965 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:59.829025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:32:59.830226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:32:59.830701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:32:59.831347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:32:59.833512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:32:59.835577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:32:59.837999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 170/358] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:59 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:32:59.841694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:59.851378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:32:59.865186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 189/359] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:32:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:32:59.866117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cb931b-0d90-43b4-a41c-ef1acef8884e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:32:59.975349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25188797-7d8a-4c35-b83c-bd1201bcdce4 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.014923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.015554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.015834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.016107 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.017060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cb931b-0d90-43b4-a41c-ef1acef8884e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.017251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:00.018738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:00.020686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 171/360] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:32:59 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:00.021466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.070529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.071495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.071955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.072505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.073803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:00.075678 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:00.077287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 190/361] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:32:59 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:00.078045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.104541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:00.163390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbfcfe-a911-44f8-b1c8-4de3e482bd77 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.177998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:00.202022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.202833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.203227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.203607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.218132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:00.225025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af6043-aef6-4db4-93f6-32cd88f70b61 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:00.226247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:00.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:00.228781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:00.230251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 172/362] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:00.231908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.284361 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.285027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.285310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.285578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.286861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:00.288375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:00.290211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 191/363] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:00.290806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.364359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b13f54-bd92-453a-b179-2084b4ce8299 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.390291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:00.410336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495e0151-b13c-4bc3-9421-7865bc8d67a5 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.410548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.411048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.411325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.411594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.425612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:00.430199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:00.431130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:33:00.431892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:00.435411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 173/364] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:00.435759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.463505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.464124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.464430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.464823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.465573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=7536b89e67c444c584e6eb0afdb7c343) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:00.465879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:00.468929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:00.469890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:00.475950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 192/365] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 260 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:00.481927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.589977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80257e42-0c56-42bf-93b6-50ab2525b124 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.608070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69476b7-686a-4c91-ba3b-973a9e1f958e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.640644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.641855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.642343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.643171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.644073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:00.644986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:00.647075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:00.652255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.653013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.653364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.653703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.660980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 174/366] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:00.662454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.667561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:00.675057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:00.675984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:00.676899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 193/367] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:00.678504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.724120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:00.745120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:00.789239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c53f5fe-4fe2-49b1-967c-5baaa4243bf3 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.789654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae06b731-d7a2-4fb1-953c-fa1ab5314cb7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.826866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.827778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.828190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:00.828502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.828798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:00.829076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:00.829319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.829417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:00.830070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] RBAC: Adding query filter params (name=reader) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:00.830365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:00.831020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=1d3187c323414f38ae8026b57e57504b) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:00.831506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:00.831764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:00.833299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:00.840032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 175/368] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:00.841174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.841805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 194/369] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:00.842808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:00.959868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b4a71c-02e0-4d6b-8c74-87d3c56e2b73 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:00.968150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:00.999972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:01.001315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.002344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.002841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.097088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da506e-dfec-4698-b2a7-5f3e0c1971bf None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:01.098331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:01.099960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:01.123784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:01.147099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:01.147724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.148005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.148278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.148949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:01.149253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:01.150653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:01.166690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 195/370] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:01.167142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:01.320956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 176/371] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:01.322123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:01.328008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f752b3-362f-4163-b016-6d57a5bb4624 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:01.343844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:01.427634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:01.428297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.428587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.428861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] PATH_INFO: `/v3/users/55dbcbf5935d48c6b17c6133159f57c7` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.431103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] RBAC: Authorizing `identity:get_user(user_id=55dbcbf5935d48c6b17c6133159f57c7)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:01.432662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:01.434658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 196/372] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:33:01 2022] GET /identity/v3/users/55dbcbf5935d48c6b17c6133159f57c7 => generated 293 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:01.435252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:01.467639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91f2bda-fb58-4e31-879d-c50259665d60 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:01.504295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:01.507084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:01.508008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.508491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.508977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.566543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfa8cbf-c37e-4ada-baf3-db66f283cc0e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:01.601081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:01.602747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:01.612309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:01.616429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c662cbe0-8e31-468a-bf91-85e2638d4559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:01.617653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662cbe0-8e31-468a-bf91-85e2638d4559 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.618661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662cbe0-8e31-468a-bf91-85e2638d4559 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.619085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662cbe0-8e31-468a-bf91-85e2638d4559 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.620799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c662cbe0-8e31-468a-bf91-85e2638d4559 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:01.623155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c662cbe0-8e31-468a-bf91-85e2638d4559 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:01.625767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 197/373] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:01 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:01.631833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2369f6f2-9846-4629-867c-94c17d845c73 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.631833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2369f6f2-9846-4629-867c-94c17d845c73 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.631833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2369f6f2-9846-4629-867c-94c17d845c73 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.632687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 198/374] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:01 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:01.634032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2369f6f2-9846-4629-867c-94c17d845c73 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:01.737820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 177/375] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:01.737820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:01.792926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2369f6f2-9846-4629-867c-94c17d845c73 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:01.818125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:01.847121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651755e0-0392-4d43-bec2-e2c05616b5f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:01.848952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651755e0-0392-4d43-bec2-e2c05616b5f1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.849453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651755e0-0392-4d43-bec2-e2c05616b5f1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.850201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651755e0-0392-4d43-bec2-e2c05616b5f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.851630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651755e0-0392-4d43-bec2-e2c05616b5f1 None admin] RBAC: Adding query filter params (role.id=35cf5ed465b441dc825c683f9e9a02a6, user.id=55dbcbf5935d48c6b17c6133159f57c7, scope.system=all) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:01.852384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651755e0-0392-4d43-bec2-e2c05616b5f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:01.853911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48691146-efb1-412c-93d3-70d753d0b18f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:01.854374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651755e0-0392-4d43-bec2-e2c05616b5f1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:01.863003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:01.875557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 199/376] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Oct 4 08:33:01 2022] GET /identity/v3/role_assignments?user.id=55dbcbf5935d48c6b17c6133159f57c7&scope.system=all&role.id=35cf5ed465b441dc825c683f9e9a02a6 => generated 510 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:01.877571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2166110b-d7f6-4276-aa3f-1c84ec35be17 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.878053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2166110b-d7f6-4276-aa3f-1c84ec35be17 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.878562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2166110b-d7f6-4276-aa3f-1c84ec35be17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.900701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:01.901437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:01.901783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:01.902122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:01.903366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:01.904863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:01.905166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-2166110b-d7f6-4276-aa3f-1c84ec35be17 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:01.906622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 178/377] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:01 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:01.906786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2166110b-d7f6-4276-aa3f-1c84ec35be17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:01.907567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:02.033745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 200/378] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:02.034972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2166110b-d7f6-4276-aa3f-1c84ec35be17 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:02.079199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455984cd-ec24-437d-b4d6-239caa2bac1f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:02.162944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2166110b-d7f6-4276-aa3f-1c84ec35be17 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:02.178835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:02.194651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:02.195307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:02.195578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:02.195845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] PATH_INFO: `/v3/projects/24e9902302e04e8e8d4caf3bfee360d5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:02.200331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] RBAC: Authorizing `identity:get_project(project_id=24e9902302e04e8e8d4caf3bfee360d5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:02.201847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:02.204310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 179/379] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:01 2022] GET /identity/v3/projects/24e9902302e04e8e8d4caf3bfee360d5 => generated 344 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:02.211407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:02.212015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:02.212291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:02.212560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:02.213400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=1d3187c323414f38ae8026b57e57504b) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:02.213867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:02.215390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:02.225648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 201/380] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:02 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:02.227120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:02.276746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:02.341085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a11306-194a-4562-aa81-7911b3337d25 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:02.393980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:02.394668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:02.394942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:02.395208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] PATH_INFO: `/v3/groups/8cf9f75b9e4a4812b0b99b044f0ea62f` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:02.402111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] RBAC: Authorizing `identity:get_group(group_id=8cf9f75b9e4a4812b0b99b044f0ea62f)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:02.403599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:02.407392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 180/381] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:02 2022] GET /identity/v3/groups/8cf9f75b9e4a4812b0b99b044f0ea62f => generated 229 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:02.412882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:02.486445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:02.575152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e93867-41f3-46fd-b5dd-ac6b9f94e342 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:02.624763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1402924-9de4-4b35-871e-9e79a1bce346 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:02.636233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:02.643122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:02.644069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:02.644573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:02.645135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:02.646042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] RBAC: Adding query filter params (group.id=8cf9f75b9e4a4812b0b99b044f0ea62f, role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=24e9902302e04e8e8d4caf3bfee360d5) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:02.646565 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:02.648184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:02.659630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 202/382] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:02 2022] GET /identity/v3/role_assignments?group.id=8cf9f75b9e4a4812b0b99b044f0ea62f&scope.project.id=24e9902302e04e8e8d4caf3bfee360d5&role.id=1922dc727b87489999cc8916eacc229e => generated 260 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:02.681806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:02.682526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:02.682824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:02.683102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:02.700350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:02.705917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:02.706907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:02.707686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:02.711623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 181/383] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:02.717636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:02.768828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:02.846345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b619a30a-42f4-4ce4-90db-124eebc9bf2e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:02.889847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:02.890554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:02.890834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:02.891102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:02.891765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:02.892061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:02.893504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:02.903353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 203/384] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:33:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:02.909059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:03.019280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29238847-f71e-45e9-a82f-c23e00a432e2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:03.058703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf89412-0436-4725-a0d3-a2c015b278ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:03.059335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf89412-0436-4725-a0d3-a2c015b278ad None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:03.059605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf89412-0436-4725-a0d3-a2c015b278ad None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:03.059871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf89412-0436-4725-a0d3-a2c015b278ad None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:03.060963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf89412-0436-4725-a0d3-a2c015b278ad None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:03.062497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf89412-0436-4725-a0d3-a2c015b278ad None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:03.064373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 182/385] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:02 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:03.070024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:03.141692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544c6ec9-e654-4c53-960c-fd3c9b443d46 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:03.142219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544c6ec9-e654-4c53-960c-fd3c9b443d46 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:03.142219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544c6ec9-e654-4c53-960c-fd3c9b443d46 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:03.144445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 183/386] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:03.149805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282943e3-2f64-42ad-b0d3-dabcd2738c94 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:03.150370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282943e3-2f64-42ad-b0d3-dabcd2738c94 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:03.150877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282943e3-2f64-42ad-b0d3-dabcd2738c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:03.180587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-282943e3-2f64-42ad-b0d3-dabcd2738c94 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:03.182302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-282943e3-2f64-42ad-b0d3-dabcd2738c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:03.210776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:03.221278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062d4f0-3071-4330-b873-f6cbb16204e3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:03.240463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:03.284166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:03.284732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:03.285047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:03.285282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:03.286478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:03.287934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:03.289744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 204/387] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:03 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:03.295481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:03.311769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 184/388] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:03.437106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511ed9db-cc9d-44c9-b59f-019a52b3393f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:03.480392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:03.481061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:03.481339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:03.481610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343/groups/0a3b295090824ef9b82d9386937f0dc3/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:03.483670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] RBAC: Authorizing `identity:create_grant(project_id=7536b89e67c444c584e6eb0afdb7c343, group_id=0a3b295090824ef9b82d9386937f0dc3, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:03.485118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:03.560734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 205/389] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Oct 4 08:33:03 2022] PUT /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343/groups/0a3b295090824ef9b82d9386937f0dc3/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:03.708313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282943e3-2f64-42ad-b0d3-dabcd2738c94 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:03.832253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282943e3-2f64-42ad-b0d3-dabcd2738c94 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:03.873935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:03.874609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:03.874890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:03.875160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:03.888078 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:03.898225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:03.899142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:03.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 185/390] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:03.905924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:03.949169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:04.052816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d3696d-906b-4971-925a-aa8b1dae780b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:04.098640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07e7dba-8247-4b39-9edf-3661f984b5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:04.099193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e7dba-8247-4b39-9edf-3661f984b5db None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:04.099477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e7dba-8247-4b39-9edf-3661f984b5db None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:04.099751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e7dba-8247-4b39-9edf-3661f984b5db None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:04.100438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e7dba-8247-4b39-9edf-3661f984b5db None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:04.100742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e7dba-8247-4b39-9edf-3661f984b5db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:04.102177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e7dba-8247-4b39-9edf-3661f984b5db None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:04.115319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 206/391] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:04.121635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:04.159592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:04.239996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcacb220-486d-4b05-9d2f-ae0a0569d456 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:04.258601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711dbe8d-d73b-4a2a-8e28-412a769e56e5 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:04.258901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711dbe8d-d73b-4a2a-8e28-412a769e56e5 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:04.259162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711dbe8d-d73b-4a2a-8e28-412a769e56e5 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:04.260629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 207/392] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:04.270767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0e2cfa-0a67-4e1d-b96d-39b0d638d21c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:04.271343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0e2cfa-0a67-4e1d-b96d-39b0d638d21c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:04.271809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0e2cfa-0a67-4e1d-b96d-39b0d638d21c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:04.282435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:04.283182 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:04.283452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:04.283730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:04.284865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:04.286316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:04.288252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 186/393] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:04 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:04.295436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:04.310250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ed0e2cfa-0a67-4e1d-b96d-39b0d638d21c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:04.311951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ed0e2cfa-0a67-4e1d-b96d-39b0d638d21c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:04.446652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 208/394] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:04.462678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:04.475120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3368d2-8642-4e8e-940e-5bf81e7c0126 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:04.515299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:04.516587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:04.516913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:04.517191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:04.519011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:04.520452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:04.526654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 187/395] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:04 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:04.528466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0e2cfa-0a67-4e1d-b96d-39b0d638d21c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:04.552956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:04.651776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0e2cfa-0a67-4e1d-b96d-39b0d638d21c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:04.697524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:04.698563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:04.699037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:04.699521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343/groups/0a3b295090824ef9b82d9386937f0dc3/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:04.708273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] RBAC: Authorizing `identity:create_grant(project_id=7536b89e67c444c584e6eb0afdb7c343, group_id=0a3b295090824ef9b82d9386937f0dc3, role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:04.710002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:04.806849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 209/396] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Oct 4 08:33:04 2022] PUT /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343/groups/0a3b295090824ef9b82d9386937f0dc3/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:04.968494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:05.094941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822b092-764f-4128-851e-1367481b7bd9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:05.134608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:05.135595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.136096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.136556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.148721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:05.153879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:05.154759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:05.155552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:05.158666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 188/397] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:05.165005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:05.247051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:05.278259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af06002-39e5-4342-a678-0ea70136891e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:05.319538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b699bd21-88c8-4f5c-828f-6f37663f0669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:05.320498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699bd21-88c8-4f5c-828f-6f37663f0669 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.320989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699bd21-88c8-4f5c-828f-6f37663f0669 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.321448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699bd21-88c8-4f5c-828f-6f37663f0669 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.322350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699bd21-88c8-4f5c-828f-6f37663f0669 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:05.322889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699bd21-88c8-4f5c-828f-6f37663f0669 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:05.324527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699bd21-88c8-4f5c-828f-6f37663f0669 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:05.338063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 210/398] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:33:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:05.343907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:05.373912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:05.483400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c0210-813b-472a-8b7d-85fffe088ab0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:05.490724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debc5c6-00f6-4639-923c-167371cc4b53 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.491003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debc5c6-00f6-4639-923c-167371cc4b53 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.491269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debc5c6-00f6-4639-923c-167371cc4b53 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.492583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 211/399] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:05.498004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079eb9b-a647-4908-a8c9-cc3b1c36c4f7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.498214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079eb9b-a647-4908-a8c9-cc3b1c36c4f7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.498561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079eb9b-a647-4908-a8c9-cc3b1c36c4f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.527493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7079eb9b-a647-4908-a8c9-cc3b1c36c4f7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:05.528709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7079eb9b-a647-4908-a8c9-cc3b1c36c4f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:05.543462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:05.544087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.544361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.544628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.545902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:05.547533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:05.549793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 189/400] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:05 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:05.557361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:05.633402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 212/401] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:05.633402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67c15a-add7-4f69-9346-284cc3533256 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.633402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67c15a-add7-4f69-9346-284cc3533256 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.633402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67c15a-add7-4f69-9346-284cc3533256 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.633402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 213/402] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:05.641912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1068c97-0229-416d-97e3-1b634a9d1a9d None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.642176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1068c97-0229-416d-97e3-1b634a9d1a9d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.642637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1068c97-0229-416d-97e3-1b634a9d1a9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.679259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d1068c97-0229-416d-97e3-1b634a9d1a9d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:05.680468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d1068c97-0229-416d-97e3-1b634a9d1a9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:05.686761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:05.722546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e3156d-3e26-4bea-aa97-781aa666d74f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:05.764314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:05.764947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.765237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.765489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.767939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:05.769570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:05.772955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 190/403] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:05 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:05.780050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:05.810767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:05.828010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 214/404] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:05.906897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d02a93-95be-4c03-bb30-a198bf4420ab None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:05.942018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:05.942812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:05.943085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:05.943353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b/groups/0a3b295090824ef9b82d9386937f0dc3/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:05.945245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] RBAC: Authorizing `identity:create_grant(project_id=1d3187c323414f38ae8026b57e57504b, group_id=0a3b295090824ef9b82d9386937f0dc3, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:05.946701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:05.994581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1068c97-0229-416d-97e3-1b634a9d1a9d None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.051831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 191/405] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Oct 4 08:33:05 2022] PUT /identity/v3/projects/1d3187c323414f38ae8026b57e57504b/groups/0a3b295090824ef9b82d9386937f0dc3/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:06.135213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1068c97-0229-416d-97e3-1b634a9d1a9d None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.184441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.185201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.185422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.185694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.191394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.209618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:06.214200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:06.215758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:06.216624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:06.217398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 215/406] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:06.227038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.310518 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6265c604-7d19-425a-8bb4-fd1a770a4a92 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.350867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:06.352449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.353088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.353361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.353624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.380623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:06.385451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:06.386319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:06.387134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 192/407] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:06.393974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.402460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94025af-523f-4f77-ad75-fe5c69d8bcab None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.449438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b31fe52-2045-4991-8089-966a298fc940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.450522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b31fe52-2045-4991-8089-966a298fc940 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.450801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b31fe52-2045-4991-8089-966a298fc940 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.451090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b31fe52-2045-4991-8089-966a298fc940 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.452537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b31fe52-2045-4991-8089-966a298fc940 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:06.452840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b31fe52-2045-4991-8089-966a298fc940 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:06.454513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b31fe52-2045-4991-8089-966a298fc940 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:06.467464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 216/408] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:06.469628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e48ea5-a199-44c1-b45a-ecb227c212db None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.469906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e48ea5-a199-44c1-b45a-ecb227c212db None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.470177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e48ea5-a199-44c1-b45a-ecb227c212db None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.472666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 217/409] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:06.473396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e48ea5-a199-44c1-b45a-ecb227c212db None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.491869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:06.522777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a52e64-f88b-49e8-98c7-b698dd23d84b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.563190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e195d3b5-afe3-4b83-94e8-74f895620ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.563984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e195d3b5-afe3-4b83-94e8-74f895620ff7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.564584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e195d3b5-afe3-4b83-94e8-74f895620ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.565035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e195d3b5-afe3-4b83-94e8-74f895620ff7 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.566348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e195d3b5-afe3-4b83-94e8-74f895620ff7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:06.566805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e195d3b5-afe3-4b83-94e8-74f895620ff7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:06.568792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e195d3b5-afe3-4b83-94e8-74f895620ff7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:06.582498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 193/410] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:06.583820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe9136-dab0-4272-93a1-b01a74259b87 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.584188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe9136-dab0-4272-93a1-b01a74259b87 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.584519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fe9136-dab0-4272-93a1-b01a74259b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.600780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e48ea5-a199-44c1-b45a-ecb227c212db None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.609975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e1fe9136-dab0-4272-93a1-b01a74259b87 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:06.611598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e1fe9136-dab0-4272-93a1-b01a74259b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:06.623341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:06.639970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.640685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.641482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.641831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] PATH_INFO: `/v3/groups/8cf9f75b9e4a4812b0b99b044f0ea62f` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.643843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] RBAC: Authorizing `identity:get_group(group_id=8cf9f75b9e4a4812b0b99b044f0ea62f)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:06.645295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:06.647229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 218/411] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/groups/8cf9f75b9e4a4812b0b99b044f0ea62f => generated 229 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:06.648030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.693539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 194/412] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:06.694979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fe9136-dab0-4272-93a1-b01a74259b87 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.831757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:06.833976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55d6b02-35fb-42be-b807-e9805018586a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.865997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fe9136-dab0-4272-93a1-b01a74259b87 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.908718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.909862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.911195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.912210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] PATH_INFO: `/v3/projects/24e9902302e04e8e8d4caf3bfee360d5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.914132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] RBAC: Authorizing `identity:get_project(project_id=24e9902302e04e8e8d4caf3bfee360d5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:06.916142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:06.920856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 195/413] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/projects/24e9902302e04e8e8d4caf3bfee360d5 => generated 344 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:06.921719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.922598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.923067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.923612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.925057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:06.926084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.928090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:06.934059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 219/414] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:06.939950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:07.017910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:07.064662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beacbc1-d88e-4846-8866-2ba15a0d9ee3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:07.085433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60f273c-ba1d-412d-ad84-2b6286e87fcc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:07.100357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:07.101053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.101431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.101790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.103182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:07.104793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:07.111885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 220/415] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:07.112865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:07.131488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:07.132669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.133089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.133898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] PATH_INFO: `/v3/projects/24e9902302e04e8e8d4caf3bfee360d5/groups/8cf9f75b9e4a4812b0b99b044f0ea62f/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.135997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] RBAC: Authorizing `identity:create_grant(project_id=24e9902302e04e8e8d4caf3bfee360d5, group_id=8cf9f75b9e4a4812b0b99b044f0ea62f, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:07.138468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:07.147267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:07.222987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990c9938-41b5-4b53-966e-78cd83a702ad None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:07.229032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 196/416] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Oct 4 08:33:06 2022] PUT /identity/v3/projects/24e9902302e04e8e8d4caf3bfee360d5/groups/8cf9f75b9e4a4812b0b99b044f0ea62f/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:07.233341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:07.296220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:07.343832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:07.344409 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.344743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.345019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.370629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bc51d-744c-4f46-bb75-f50ef894d4cb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:07.375517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:07.381608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:07.382537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:07.383324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:07.390531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 221/417] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:07.399471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:07.418381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434c9178-2b25-42fc-b2ca-3aa81e66fade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:07.419267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434c9178-2b25-42fc-b2ca-3aa81e66fade None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.420038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434c9178-2b25-42fc-b2ca-3aa81e66fade None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.420432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434c9178-2b25-42fc-b2ca-3aa81e66fade None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b/groups/0a3b295090824ef9b82d9386937f0dc3/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.424472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434c9178-2b25-42fc-b2ca-3aa81e66fade None admin] RBAC: Authorizing `identity:create_grant(project_id=1d3187c323414f38ae8026b57e57504b, group_id=0a3b295090824ef9b82d9386937f0dc3, role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:07.426832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434c9178-2b25-42fc-b2ca-3aa81e66fade None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:07.502915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 197/418] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Oct 4 08:33:07 2022] PUT /identity/v3/projects/1d3187c323414f38ae8026b57e57504b/groups/0a3b295090824ef9b82d9386937f0dc3/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:07.538056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f066a0a4-f437-48fd-89cd-dc2162a04f39 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:07.584727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:07.585459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.585853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.586132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.586911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] RBAC: Adding query filter params (name=member) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:07.587216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:07.588596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:07.600756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 222/419] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:33:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:07.601369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977c09d-9afc-42de-a8eb-feb41916587c None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.601659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977c09d-9afc-42de-a8eb-feb41916587c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.601931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977c09d-9afc-42de-a8eb-feb41916587c None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.606496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 198/420] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:07.608625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:07.615972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f9a88d-a656-4e8a-b33e-9797e7b4155a None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.616247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f9a88d-a656-4e8a-b33e-9797e7b4155a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.616526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f9a88d-a656-4e8a-b33e-9797e7b4155a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.662591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-82f9a88d-a656-4e8a-b33e-9797e7b4155a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:07.663833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-82f9a88d-a656-4e8a-b33e-9797e7b4155a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:07.667874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:07.738183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4676ddcb-d0e6-45b4-b5f8-caee64478ae8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:07.760160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:07.777523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 199/421] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:07.786673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:07.788257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.788531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.788849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.861573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:07.862975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:07.910568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:07.965649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 223/422] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:07.972176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f9a88d-a656-4e8a-b33e-9797e7b4155a None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:08.130717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f9a88d-a656-4e8a-b33e-9797e7b4155a None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:08.176990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:08.177634 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.177912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.178176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.179353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:08.180775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:08.182619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 200/423] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:07 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:08.190268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:08.203329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:08.307272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c02cf6b-0eae-4598-afe5-6f1f0569cb58 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:08.310769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:08.355165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:08.355875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.356156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.356414 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.357552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:08.359032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:08.361036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 224/424] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:08 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:08.366705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:08.372286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebea9268-775b-4506-bc7d-21dd6ef78292 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:08.410236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:08.428886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:08.429525 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.429800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.430067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.451660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:08.459639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:08.460562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:08.461344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:08.462907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 201/425] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:08.468544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:08.557834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1bea22-ce91-47af-9bda-6cbc4995616a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:08.608646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbacd000-2b43-4653-bba7-6a583fac02a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:08.609264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbacd000-2b43-4653-bba7-6a583fac02a7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.609539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbacd000-2b43-4653-bba7-6a583fac02a7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.609809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbacd000-2b43-4653-bba7-6a583fac02a7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.610559 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbacd000-2b43-4653-bba7-6a583fac02a7 None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=7536b89e67c444c584e6eb0afdb7c343) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:08.610864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbacd000-2b43-4653-bba7-6a583fac02a7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:08.616575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbacd000-2b43-4653-bba7-6a583fac02a7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:08.638028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 225/426] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:08 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 611 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:08.669782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a82bd-ac79-4e1d-b85d-bae10547a984 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:08.684073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e7f56-3ee4-4488-853a-609749156521 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.684408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e7f56-3ee4-4488-853a-609749156521 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.684619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e7f56-3ee4-4488-853a-609749156521 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.686050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 226/427] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:08.697904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf4289-69cd-4562-84ba-30903979dd13 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.698230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf4289-69cd-4562-84ba-30903979dd13 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.698463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabf4289-69cd-4562-84ba-30903979dd13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.698776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:08.731645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-aabf4289-69cd-4562-84ba-30903979dd13 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:08.733774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-aabf4289-69cd-4562-84ba-30903979dd13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:08.743938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:08.744968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.745459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.745959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.747961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:08.748539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:08.754852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:08.776010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 202/428] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:08.780701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:08.883877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 227/429] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:08.935989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97047329-9d43-422d-86a0-ae5cdccc9aec None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:08.973878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:08.974516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.974791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.975060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.023762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:09.046224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:09.057975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:09.150696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 203/430] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:33:09.155989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabf4289-69cd-4562-84ba-30903979dd13 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:09.168100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:09.196174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:09.289835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabf4289-69cd-4562-84ba-30903979dd13 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:09.300989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:09.325423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb2d8aa-545d-47df-b6d8-237cd48860dd None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:09.338696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:09.339497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.339795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.340198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] PATH_INFO: `/v3/projects/7536b89e67c444c584e6eb0afdb7c343` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.342204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] RBAC: Authorizing `identity:get_project(project_id=7536b89e67c444c584e6eb0afdb7c343)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:09.344208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:09.348546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 228/431] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:09 2022] GET /identity/v3/projects/7536b89e67c444c584e6eb0afdb7c343 => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:09.353877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:09.364955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:09.365887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.366334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.366831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] PATH_INFO: `/v3/roles/member` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.378742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:09.384546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:09.385438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:33:09.386198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:09.388883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 204/432] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:09.395655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:09.442864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:09.472196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591d04e7-e3de-4115-a3e1-27006bdbc9b8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:09.509615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffe98c8-00f2-4127-a475-6f3048775945 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:09.513458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:09.513868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.514236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.516286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.518011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:09.519814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:09.522449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 229/433] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:09 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:09.527359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:09.551970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:09.552608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.552886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.553158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.553825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] RBAC: Adding query filter params (name=member) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:09.554143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:09.557434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:09.559974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:09.567402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 205/434] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:33:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:09.573098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:09.641918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc048c4-498f-4826-99b3-8eccb2349aa1 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:09.682926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c819be-7ad8-43ac-9872-0e25cd783d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:09.683916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c819be-7ad8-43ac-9872-0e25cd783d6f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.684388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c819be-7ad8-43ac-9872-0e25cd783d6f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.684845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c819be-7ad8-43ac-9872-0e25cd783d6f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.685872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c819be-7ad8-43ac-9872-0e25cd783d6f None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=7536b89e67c444c584e6eb0afdb7c343) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:09.686499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c819be-7ad8-43ac-9872-0e25cd783d6f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:09.688370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c819be-7ad8-43ac-9872-0e25cd783d6f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:09.696141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984317a-31a5-42e4-a4b2-ffe5e8964bec None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:09.697247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 230/435] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:09 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=7536b89e67c444c584e6eb0afdb7c343&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 611 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:09.739945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:09.740859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.741316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.741735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.826086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:09.831505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:09.833147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:09.879320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf02c5f-3f11-46fe-b8e3-618079c82c78 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.879989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf02c5f-3f11-46fe-b8e3-618079c82c78 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.880464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf02c5f-3f11-46fe-b8e3-618079c82c78 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.882336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 231/436] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:09.888158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06aa7e5-e78b-43de-8865-33dfd30ef072 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:09.888747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06aa7e5-e78b-43de-8865-33dfd30ef072 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:09.889240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06aa7e5-e78b-43de-8865-33dfd30ef072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:09.909355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c06aa7e5-e78b-43de-8865-33dfd30ef072 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:09.910580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c06aa7e5-e78b-43de-8865-33dfd30ef072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:09.913530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:09.952305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 206/437] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:33:09.958665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.000177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 232/438] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:10.075180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:10.080153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceab67da-12db-4ed5-82b2-1abdf05d1aaf None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.117221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f09a0b-9dfd-48b2-9537-4573665b2e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.118536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f09a0b-9dfd-48b2-9537-4573665b2e71 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.119041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f09a0b-9dfd-48b2-9537-4573665b2e71 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.119511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f09a0b-9dfd-48b2-9537-4573665b2e71 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.120852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f09a0b-9dfd-48b2-9537-4573665b2e71 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:10.122539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f09a0b-9dfd-48b2-9537-4573665b2e71 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:10.124669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 207/439] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:09 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:10.130112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06aa7e5-e78b-43de-8865-33dfd30ef072 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.139987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df96ee7a-5e97-4d98-8657-200a310a522e None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.140623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df96ee7a-5e97-4d98-8657-200a310a522e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.140947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df96ee7a-5e97-4d98-8657-200a310a522e None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.143092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 208/440] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:10.151788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f599f03d-c783-4087-8053-0d030ed335d1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.152066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f599f03d-c783-4087-8053-0d030ed335d1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.152340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f599f03d-c783-4087-8053-0d030ed335d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.172973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f599f03d-c783-4087-8053-0d030ed335d1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:10.174809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f599f03d-c783-4087-8053-0d030ed335d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:10.258939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 209/441] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:10.266137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06aa7e5-e78b-43de-8865-33dfd30ef072 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.283429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f599f03d-c783-4087-8053-0d030ed335d1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.302699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.303325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.303600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.303869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.304970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:10.306426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:10.308146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 233/442] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:10.313603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.322351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:10.381559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:10.387209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f599f03d-c783-4087-8053-0d030ed335d1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.423008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905ace76-4e6f-4e76-a540-42e21f579888 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.425609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.426839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.426839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.426839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.441594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:10.446587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:10.448340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:10.449336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 210/443] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:10.453444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.461950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.462590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.462871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.463141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.463870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=40b933f8a46e4be9bd70366ec14885e0, scope.project.id=1d3187c323414f38ae8026b57e57504b) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:10.464173 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:10.465688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:10.471792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 234/444] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=40b933f8a46e4be9bd70366ec14885e0 => generated 611 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:10.529625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.570377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf922293-c5ca-4553-a835-00927f147673 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.608892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:10.613631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.614267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.614578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.614855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.615545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:10.615896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:10.617258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:10.627419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 211/445] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:10.633815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.641999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688691e8-9f6b-4c30-90e0-c2acbeaae89a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.656680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:10.681571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.682857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.682857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.682857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.698918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:10.704283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:10.705398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:10.706385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 235/446] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:10.713421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.761327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7fbf82-7fcf-4160-b81c-8b3736d820ec None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.802594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.803135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.803410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.803673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.841371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bcd8f6-ce17-4c0a-a7c1-b320c6d86e75 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:10.876458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:10.882631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:10.883238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:10.883510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:10.883775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:10.884488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:10.884779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:10.886128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:10.886280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:10.887758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:10.894503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 236/447] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:10.900257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.905919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:10.987307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 212/448] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:33:10.992877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:10.998382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0076a7a6-16d8-4186-83d0-a27a8b1fca5e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.032241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.032915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.033195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.033607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.088886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d357a3-d4ed-4693-8ffc-38644da4d63d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.091482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:11.100650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.102227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.104895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:11.121927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.122577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.122855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.123124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] PATH_INFO: `/v3/projects/24e9902302e04e8e8d4caf3bfee360d5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.124227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] RBAC: Authorizing `identity:get_project(project_id=24e9902302e04e8e8d4caf3bfee360d5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.125637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.128078 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 213/449] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/projects/24e9902302e04e8e8d4caf3bfee360d5 => generated 344 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:11.133664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:11.199270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 237/450] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:11.204827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:11.232734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e53d08-3f5a-42f9-8e91-034808758d73 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.268321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.268940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.269216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.269487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] PATH_INFO: `/v3/groups/8cf9f75b9e4a4812b0b99b044f0ea62f` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.270630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] RBAC: Authorizing `identity:get_group(group_id=8cf9f75b9e4a4812b0b99b044f0ea62f)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.272068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.273603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 214/451] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:11 2022] GET /identity/v3/groups/8cf9f75b9e4a4812b0b99b044f0ea62f => generated 229 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:11.279407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:11.295279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:11.304192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfa51e7-b506-4b08-b6a8-9c8cab5c118a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.335353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:11.339256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.339864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.340148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.340441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] PATH_INFO: `/v3/projects/1d3187c323414f38ae8026b57e57504b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.341583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] RBAC: Authorizing `identity:get_project(project_id=1d3187c323414f38ae8026b57e57504b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.343053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.344642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 238/452] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:33:11 2022] GET /identity/v3/projects/1d3187c323414f38ae8026b57e57504b => generated 302 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:11.350286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:11.379844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c502b31f-8e63-46d2-ac42-06f3f9854177 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.414460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.415057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.415331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.415600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.416310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] RBAC: Adding query filter params (group.id=8cf9f75b9e4a4812b0b99b044f0ea62f, role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=24e9902302e04e8e8d4caf3bfee360d5) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:11.416612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.418139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.424284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 215/453] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:11 2022] GET /identity/v3/role_assignments?group.id=8cf9f75b9e4a4812b0b99b044f0ea62f&scope.project.id=24e9902302e04e8e8d4caf3bfee360d5&role.id=1922dc727b87489999cc8916eacc229e => generated 611 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:11.452591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1156b4-373b-4153-be6a-db2a8bb8ff87 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.491671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e9110d-52cd-422c-9cb1-b72c41b03cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.492314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e9110d-52cd-422c-9cb1-b72c41b03cd4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.492588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e9110d-52cd-422c-9cb1-b72c41b03cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.492858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e9110d-52cd-422c-9cb1-b72c41b03cd4 None admin] PATH_INFO: `/v3/groups/0a3b295090824ef9b82d9386937f0dc3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.494277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e9110d-52cd-422c-9cb1-b72c41b03cd4 None admin] RBAC: Authorizing `identity:get_group(group_id=0a3b295090824ef9b82d9386937f0dc3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.495757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e9110d-52cd-422c-9cb1-b72c41b03cd4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.497369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 239/454] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Oct 4 08:33:11 2022] GET /identity/v3/groups/0a3b295090824ef9b82d9386937f0dc3 => generated 226 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:11.504001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:11.607885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5d71ff-6c1a-4b2d-8c2f-cd14342e260a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.625613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:11.642945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263d4be9-21d0-42fa-8ec2-cc1032363f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.643599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263d4be9-21d0-42fa-8ec2-cc1032363f40 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.643875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263d4be9-21d0-42fa-8ec2-cc1032363f40 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.644145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263d4be9-21d0-42fa-8ec2-cc1032363f40 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.644880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263d4be9-21d0-42fa-8ec2-cc1032363f40 None admin] RBAC: Adding query filter params (group.id=0a3b295090824ef9b82d9386937f0dc3, role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=1d3187c323414f38ae8026b57e57504b) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:11.645193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263d4be9-21d0-42fa-8ec2-cc1032363f40 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.646752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263d4be9-21d0-42fa-8ec2-cc1032363f40 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.652490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 216/455] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Oct 4 08:33:11 2022] GET /identity/v3/role_assignments?group.id=0a3b295090824ef9b82d9386937f0dc3&scope.project.id=1d3187c323414f38ae8026b57e57504b&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:14.396399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c543f-ba3d-423f-9d66-ca423d4b759f None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:14.397193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c543f-ba3d-423f-9d66-ca423d4b759f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:14.397684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c543f-ba3d-423f-9d66-ca423d4b759f None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:14.399290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 240/456] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:14.404524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94c074-51c7-4472-bda4-4ad85d0efd1c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:14.404919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94c074-51c7-4472-bda4-4ad85d0efd1c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:14.405108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94c074-51c7-4472-bda4-4ad85d0efd1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:14.431780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1e94c074-51c7-4472-bda4-4ad85d0efd1c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:14.432973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1e94c074-51c7-4472-bda4-4ad85d0efd1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:14.535825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 217/457] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:14.698761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246f8e25-5799-4039-833c-15fa0707435b None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:14.698950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246f8e25-5799-4039-833c-15fa0707435b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:14.699224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246f8e25-5799-4039-833c-15fa0707435b None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:14.700508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 241/458] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:14.706952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8758ad39-79d0-4d21-9a9d-2b64b2808369 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:14.707234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8758ad39-79d0-4d21-9a9d-2b64b2808369 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:14.707507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8758ad39-79d0-4d21-9a9d-2b64b2808369 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:14.732723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8758ad39-79d0-4d21-9a9d-2b64b2808369 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:14.733904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8758ad39-79d0-4d21-9a9d-2b64b2808369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:14.782053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774de5d5-ea6c-4a51-b838-f90ed4981f70 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:14.783125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774de5d5-ea6c-4a51-b838-f90ed4981f70 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:14.783671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774de5d5-ea6c-4a51-b838-f90ed4981f70 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:14.785564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 242/459] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:14.790845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a573c-e881-4a4f-8bc2-6c1ad72d9a04 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:14.791363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a573c-e881-4a4f-8bc2-6c1ad72d9a04 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:14.791830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a573c-e881-4a4f-8bc2-6c1ad72d9a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:14.813497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-223a573c-e881-4a4f-8bc2-6c1ad72d9a04 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:14.814766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-223a573c-e881-4a4f-8bc2-6c1ad72d9a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:14.850794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 218/460] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:14.861555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:14.928224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 243/461] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:14.979650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8758ad39-79d0-4d21-9a9d-2b64b2808369 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.037994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:15.111727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8758ad39-79d0-4d21-9a9d-2b64b2808369 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:15.156778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:15.158289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.158954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.159448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.171881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:15.174248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:15.175505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a573c-e881-4a4f-8bc2-6c1ad72d9a04 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:15.191039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:15.192006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:15.192896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:15.192896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:15.192896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:15.194927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 219/462] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:15.200620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.351257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a573c-e881-4a4f-8bc2-6c1ad72d9a04 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:15.369346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:15.379313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf7694a-9500-4aa5-8c83-2a99d4d7a045 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:15.406313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfad22d3-09c4-42a6-88b2-dae4ce34fa50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:15.406989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfad22d3-09c4-42a6-88b2-dae4ce34fa50 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.407264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfad22d3-09c4-42a6-88b2-dae4ce34fa50 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.408207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfad22d3-09c4-42a6-88b2-dae4ce34fa50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.419101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfad22d3-09c4-42a6-88b2-dae4ce34fa50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:15.420549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfad22d3-09c4-42a6-88b2-dae4ce34fa50 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:15.424852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:15.425471 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.425746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.426012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.426692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:15.426995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:15.428804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-dfad22d3-09c4-42a6-88b2-dae4ce34fa50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:15.436843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:15.437712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:15.439563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:15.439563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:15.439563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:15.439563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 244/463] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:15.440679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1275ff5-0899-43d5-81d2-12e2cb7a0b05 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.441075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1275ff5-0899-43d5-81d2-12e2cb7a0b05 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.441343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1275ff5-0899-43d5-81d2-12e2cb7a0b05 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.442908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 245/464] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:15.443570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 220/465] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:15.445365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.448046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1275ff5-0899-43d5-81d2-12e2cb7a0b05 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.461756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:15.557725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1275ff5-0899-43d5-81d2-12e2cb7a0b05 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:15.558718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183e8f90-4a43-4b98-94ec-8a3ff40c3a4f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:15.592855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fb31e6-6ca0-4b2a-9b85-fe72761bae12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:15.593489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fb31e6-6ca0-4b2a-9b85-fe72761bae12 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.593771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fb31e6-6ca0-4b2a-9b85-fe72761bae12 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.594044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fb31e6-6ca0-4b2a-9b85-fe72761bae12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.600830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:15.601790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.602302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.602667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.603455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:15.603729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:15.605162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:15.606130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fb31e6-6ca0-4b2a-9b85-fe72761bae12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:15.607625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53fb31e6-6ca0-4b2a-9b85-fe72761bae12 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:15.617905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 221/466] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:15.619303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-53fb31e6-6ca0-4b2a-9b85-fe72761bae12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:15.622813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:15.623677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:15.624485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:15.624485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:15.624485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:15.624485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 246/467] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:15.625120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc3238-1c59-4e0c-807a-d94f35461451 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.625408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc3238-1c59-4e0c-807a-d94f35461451 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.625677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc3238-1c59-4e0c-807a-d94f35461451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.642952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:15.644851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-73cc3238-1c59-4e0c-807a-d94f35461451 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:15.646338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-73cc3238-1c59-4e0c-807a-d94f35461451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:15.707058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:15.725594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 247/468] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:15.725594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc3238-1c59-4e0c-807a-d94f35461451 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.739062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fa8ac1-65e2-4b64-a1cb-74822683b7e8 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:15.784080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:15.784698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.784979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.785254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.785921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:15.787693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:15.823497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc3238-1c59-4e0c-807a-d94f35461451 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:15.858761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:15.859317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:15.859594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:15.859866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:15.860503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:15.862605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:15.931954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 222/469] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:33:15 2022] POST /identity/v3/users => generated 246 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:15.932870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:15.990011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 248/470] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:33:15 2022] POST /identity/v3/users => generated 248 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:16.008986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:16.058736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:16.094449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:16.103602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6a088a-2ba1-4c2a-bb1f-f4aaabbc52dc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:16.152086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349baadb-02a4-4239-8a7e-1b0c46106f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:16.153410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349baadb-02a4-4239-8a7e-1b0c46106f7c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.153833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349baadb-02a4-4239-8a7e-1b0c46106f7c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.154103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349baadb-02a4-4239-8a7e-1b0c46106f7c None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.154889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349baadb-02a4-4239-8a7e-1b0c46106f7c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:16.155193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349baadb-02a4-4239-8a7e-1b0c46106f7c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:16.156663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349baadb-02a4-4239-8a7e-1b0c46106f7c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:16.160328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2ecc14-839b-4896-a61a-a76a40b1df18 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:16.171290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 223/471] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:16.178157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b027bb61-8b23-43c0-9b1e-63e59c2fc6ab None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.178157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b027bb61-8b23-43c0-9b1e-63e59c2fc6ab None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.178157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b027bb61-8b23-43c0-9b1e-63e59c2fc6ab None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.178157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 224/472] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:16.178157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b027bb61-8b23-43c0-9b1e-63e59c2fc6ab None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:16.211375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329d0247-b00e-43ff-bd4a-a7c78e9a350d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:16.212302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d0247-b00e-43ff-bd4a-a7c78e9a350d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.212588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d0247-b00e-43ff-bd4a-a7c78e9a350d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.212861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d0247-b00e-43ff-bd4a-a7c78e9a350d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.227293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329d0247-b00e-43ff-bd4a-a7c78e9a350d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:16.228789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329d0247-b00e-43ff-bd4a-a7c78e9a350d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-329d0247-b00e-43ff-bd4a-a7c78e9a350d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:16.248759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:16.249703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:16.250564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:16.250564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:16.250564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:16.250731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 249/473] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:16.252429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bae95a1-94b6-434c-b732-58a3041866cd None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.252706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bae95a1-94b6-434c-b732-58a3041866cd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.253137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bae95a1-94b6-434c-b732-58a3041866cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.278841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9bae95a1-94b6-434c-b732-58a3041866cd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:16.279788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9bae95a1-94b6-434c-b732-58a3041866cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:16.312367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b027bb61-8b23-43c0-9b1e-63e59c2fc6ab None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:16.347461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:16.354166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:16.354811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.355135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.355404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.356055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:16.357491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:16.374501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 250/474] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:16.375828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bae95a1-94b6-434c-b732-58a3041866cd None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:16.413629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:16.491807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bae95a1-94b6-434c-b732-58a3041866cd None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:16.493502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 225/475] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:33:16 2022] POST /identity/v3/users => generated 248 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:16.544399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:16.545475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.545784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.546064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.546751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:16.547053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:16.548532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:16.564592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 251/476] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:16.570455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:16.704332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:16.744344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c70bf36-2f19-4ad7-89d5-5ed7b9ab6730 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:16.788388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:16.789906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.790452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.790859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.791673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:16.793519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:16.826158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:16.843329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871ef7e-583c-4d0d-a9e0-ffcba3ba4937 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:16.884967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:16.885595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:16.885869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:16.886142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:16.886884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:16.887348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 226/477] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:33:16 2022] POST /identity/v3/users => generated 249 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:16.888230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:16.935371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.sql.core [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 08:33:16.935371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 04 08:33:16.935371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [parameters: {'id': 'c007e4994373494faea5bdc438a7da77', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 04 08:33:16.935371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=88203) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application result.read() Oct 04 08:33:16.956751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return ref.to_dict() Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:33:16.957846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application yield resource Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise self.value Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application session.commit() Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 08:33:16.959657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self.session.flush() Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application compat.raise_( Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 08:33:16.960515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:33:16.961355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application result.read() Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:33:16.962125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [parameters: {'id': 'c007e4994373494faea5bdc438a7da77', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:16.964357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 04 08:33:16.966964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:16.969052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 252/478] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Oct 4 08:33:16 2022] POST /identity/v3/roles => generated 144 bytes in 254 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:16.969183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:17.042592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:17.127898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de5eeed-b3af-43d4-b2f3-1269445ea491 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:17.175338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff16050-0af4-44bc-bdfa-2abc804d7906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:17.176342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff16050-0af4-44bc-bdfa-2abc804d7906 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:17.176814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff16050-0af4-44bc-bdfa-2abc804d7906 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:17.177294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff16050-0af4-44bc-bdfa-2abc804d7906 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:17.192725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff16050-0af4-44bc-bdfa-2abc804d7906 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-aff16050-0af4-44bc-bdfa-2abc804d7906 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:17.198321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:17.199366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:33:17.200296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:17.202511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 227/479] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:17.208174 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:17.327435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6042aaf7-531e-4e27-90d8-cb65c46a5e69 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:17.369779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fbdb51-93c5-45e2-90af-4337bf9034ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:17.370399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fbdb51-93c5-45e2-90af-4337bf9034ce None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:17.370708 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fbdb51-93c5-45e2-90af-4337bf9034ce None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:17.370980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fbdb51-93c5-45e2-90af-4337bf9034ce None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:17.371637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fbdb51-93c5-45e2-90af-4337bf9034ce None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:17.371990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fbdb51-93c5-45e2-90af-4337bf9034ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:17.373479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fbdb51-93c5-45e2-90af-4337bf9034ce None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:17.386565 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 253/480] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:18.487926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c48585-e147-4e3b-a813-4482bdcc73c8 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:18.488447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c48585-e147-4e3b-a813-4482bdcc73c8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:18.488668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c48585-e147-4e3b-a813-4482bdcc73c8 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:18.490183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 228/481] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:18.496568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4795a13-b80b-4638-a84c-51de760d90b3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:18.497106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4795a13-b80b-4638-a84c-51de760d90b3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:18.497106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4795a13-b80b-4638-a84c-51de760d90b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:18.527419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a4795a13-b80b-4638-a84c-51de760d90b3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:18.528621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a4795a13-b80b-4638-a84c-51de760d90b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:18.541114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:18.624981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 254/482] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:18.626149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c8cd94-c6f7-41b1-99de-b5380dc2a5db None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:18.626887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c8cd94-c6f7-41b1-99de-b5380dc2a5db None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:18.627375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c8cd94-c6f7-41b1-99de-b5380dc2a5db None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:18.629056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 229/483] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:18.636476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55a9771-4bb8-463f-926a-1794234bc863 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:18.636763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55a9771-4bb8-463f-926a-1794234bc863 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:18.637038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55a9771-4bb8-463f-926a-1794234bc863 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:18.690071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b55a9771-4bb8-463f-926a-1794234bc863 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:18.691186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b55a9771-4bb8-463f-926a-1794234bc863 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:18.783936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 255/484] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:19.038771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c8cd94-c6f7-41b1-99de-b5380dc2a5db None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.140164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55a9771-4bb8-463f-926a-1794234bc863 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.162811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c8cd94-c6f7-41b1-99de-b5380dc2a5db None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:19.165217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:19.227489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:19.228187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.228467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.228734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.259350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:19.264154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:19.265002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:19.265757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 230/485] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:19.267956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:19.275366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.291906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55a9771-4bb8-463f-926a-1794234bc863 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:19.330399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:19.331170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.331344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.331593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] PATH_INFO: `/v3/roles/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.342139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:19.350943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:19.351829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:19.352599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 256/486] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:19.354009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.458582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29502c71-68b5-4c7b-a452-99c038122fed None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:19.505106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:19.505727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.505979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.506249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.506933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:19.507237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:19.508618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:19.514021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:19.520101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ee57be-b8c8-46b8-a009-7e400f76a63a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:19.533655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 257/487] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:19.539453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.588281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:19.588802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.589079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.589345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.590009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:19.590308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:19.600008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:19.607889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 231/488] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:19.613953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.670393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5d0c9b-1d47-482e-882f-3cc610852662 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:19.682093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:19.706685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591a2fbf-e796-4f5d-ac27-7d5db2e2c0dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:19.707334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a2fbf-e796-4f5d-ac27-7d5db2e2c0dd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.707616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a2fbf-e796-4f5d-ac27-7d5db2e2c0dd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.707885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a2fbf-e796-4f5d-ac27-7d5db2e2c0dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.718821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591a2fbf-e796-4f5d-ac27-7d5db2e2c0dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:19.720481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591a2fbf-e796-4f5d-ac27-7d5db2e2c0dd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-591a2fbf-e796-4f5d-ac27-7d5db2e2c0dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:19.737121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:19.738278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:19.739353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:19.739353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:19.739353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:19.739353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 258/489] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:19.740098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d888570-da4e-4e2c-88cd-1419dca6618c None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.740381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d888570-da4e-4e2c-88cd-1419dca6618c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.740652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d888570-da4e-4e2c-88cd-1419dca6618c None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.742117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 259/490] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:19.749541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0591ae-e951-4d0d-b873-1fd40ba3062b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.749815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0591ae-e951-4d0d-b873-1fd40ba3062b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.750079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0591ae-e951-4d0d-b873-1fd40ba3062b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.770910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-fc0591ae-e951-4d0d-b873-1fd40ba3062b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:19.772215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fc0591ae-e951-4d0d-b873-1fd40ba3062b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:19.774451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9a2a26-6916-46d7-8a3e-49bacf4830b5 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:19.815190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:19.815879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.816154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.816420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.825949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:19.829951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:19.832099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:19.849377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:19.850455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:19.851410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:19.851410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:19.851410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:19.853110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 232/491] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:19.854061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.857568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 260/492] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:19.859714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0591ae-e951-4d0d-b873-1fd40ba3062b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:19.946388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:19.970032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca5e657-2d6a-4e6d-9dce-8ba6f6eac674 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.002452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0591ae-e951-4d0d-b873-1fd40ba3062b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.007888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17ed5ae-fb38-45e1-bf80-000439cea783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.008573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17ed5ae-fb38-45e1-bf80-000439cea783 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.008923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17ed5ae-fb38-45e1-bf80-000439cea783 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.009306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17ed5ae-fb38-45e1-bf80-000439cea783 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.010019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17ed5ae-fb38-45e1-bf80-000439cea783 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:20.010392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17ed5ae-fb38-45e1-bf80-000439cea783 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.012298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17ed5ae-fb38-45e1-bf80-000439cea783 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.026225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 233/493] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:20.028057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d09c82e-488f-4989-83f1-4f8c42113123 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.028428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d09c82e-488f-4989-83f1-4f8c42113123 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.028805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d09c82e-488f-4989-83f1-4f8c42113123 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.030354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 234/494] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:20.035812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d09c82e-488f-4989-83f1-4f8c42113123 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.044649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c936a5-b02d-43e7-a43f-ba0548743eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.045972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c936a5-b02d-43e7-a43f-ba0548743eb4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.046466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c936a5-b02d-43e7-a43f-ba0548743eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.046898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c936a5-b02d-43e7-a43f-ba0548743eb4 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.048663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c936a5-b02d-43e7-a43f-ba0548743eb4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:20.049120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c936a5-b02d-43e7-a43f-ba0548743eb4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.051372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c936a5-b02d-43e7-a43f-ba0548743eb4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.067328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 261/495] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:20.069039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddbf2be-a0b1-4554-8a5b-8fe3b79d274e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.069905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddbf2be-a0b1-4554-8a5b-8fe3b79d274e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.070322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddbf2be-a0b1-4554-8a5b-8fe3b79d274e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.093550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7ddbf2be-a0b1-4554-8a5b-8fe3b79d274e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:20.095497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7ddbf2be-a0b1-4554-8a5b-8fe3b79d274e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:20.145126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:20.160416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d09c82e-488f-4989-83f1-4f8c42113123 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.188221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 262/496] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:20.189737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddbf2be-a0b1-4554-8a5b-8fe3b79d274e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.207303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.208592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.209464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.209929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.213994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:20.219997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.222603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:20.240679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:20.241647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:20.242611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 235/497] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:20.243921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.297943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddbf2be-a0b1-4554-8a5b-8fe3b79d274e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.332697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70983b8-9601-4838-8989-61098afb217c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.333323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70983b8-9601-4838-8989-61098afb217c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.333597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70983b8-9601-4838-8989-61098afb217c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.333864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70983b8-9601-4838-8989-61098afb217c None admin] PATH_INFO: `/v3/users/glance` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.340807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70983b8-9601-4838-8989-61098afb217c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.342249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70983b8-9601-4838-8989-61098afb217c None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-e70983b8-9601-4838-8989-61098afb217c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:20.355563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:20.356629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:20.357506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 263/498] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/users/glance => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:20.358037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70983b8-9601-4838-8989-61098afb217c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.366165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7c12d2-8673-4122-8ba0-5930032d74e7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.395664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:20.406259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfaf85e-3a7a-45ff-a3da-a8a648ccd2b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.406908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfaf85e-3a7a-45ff-a3da-a8a648ccd2b1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.407180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfaf85e-3a7a-45ff-a3da-a8a648ccd2b1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.408362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfaf85e-3a7a-45ff-a3da-a8a648ccd2b1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.418948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfaf85e-3a7a-45ff-a3da-a8a648ccd2b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-3bfaf85e-3a7a-45ff-a3da-a8a648ccd2b1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:20.425423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:20.426299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:20.427098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 236/499] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:20.428224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c64eda5-2ed9-470d-b822-54f00d8b5f1d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.428502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c64eda5-2ed9-470d-b822-54f00d8b5f1d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.428674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c64eda5-2ed9-470d-b822-54f00d8b5f1d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.434165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 237/500] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:20.434165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c64eda5-2ed9-470d-b822-54f00d8b5f1d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.481628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70983b8-9601-4838-8989-61098afb217c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.487997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:20.515352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.515960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.516236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.516514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.517258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:20.517554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.519020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.532273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c64eda5-2ed9-470d-b822-54f00d8b5f1d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.555764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 264/501] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:20.556587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.566950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4707633e-9a39-498d-932e-0d17ce0b31c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.567549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4707633e-9a39-498d-932e-0d17ce0b31c2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.567827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4707633e-9a39-498d-932e-0d17ce0b31c2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.568099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4707633e-9a39-498d-932e-0d17ce0b31c2 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.568824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4707633e-9a39-498d-932e-0d17ce0b31c2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:20.569126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4707633e-9a39-498d-932e-0d17ce0b31c2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.570581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4707633e-9a39-498d-932e-0d17ce0b31c2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.607415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 238/502] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:20.608632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6a45d2-3472-4899-9f97-7f3cc58b8337 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.608916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6a45d2-3472-4899-9f97-7f3cc58b8337 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.609179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6a45d2-3472-4899-9f97-7f3cc58b8337 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.631717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-2f6a45d2-3472-4899-9f97-7f3cc58b8337 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:20.632936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2f6a45d2-3472-4899-9f97-7f3cc58b8337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:20.648595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:20.660842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31427372-ec6f-4b75-ba33-cef367622d59 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.698658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.699264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.699537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.699804 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.700219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 239/503] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:20.700457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:20.700750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.701059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6a45d2-3472-4899-9f97-7f3cc58b8337 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.702123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.719515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 265/504] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:20.720971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.753245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:20.808781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6a45d2-3472-4899-9f97-7f3cc58b8337 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.826430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db92ae2-0b33-47dc-b262-fec34a1a1db1 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.848460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.848993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.849268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.849534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.860345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:20.863132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.863740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.864013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.864280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:20.866527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:20.867386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:20.868403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 240/505] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:20.869044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.891619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:20.951930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.953316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.982810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f00a8b7-e780-4b79-91b3-fbc4a6b96040 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.986468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.018492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.019089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.019365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.019632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.058890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 266/506] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:21.059741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.093291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.094788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.125114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.181270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0c8011-dded-42c3-9d99-79dfef4dd060 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.225646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.225837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 241/507] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:21.226208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.226508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.226629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.226781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.231726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.242431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.248114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:21.265647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:21.268492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:21.269313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:21.269313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.269313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:21.275090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 267/508] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:21.276126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.354586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259756c-e29a-4ba2-ba8a-b5637e052184 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.401333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.401751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9509abf-90a2-479b-9277-4122b195940e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.401927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.402195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.402482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.403216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:21.403515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.403905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.405667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.414398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 242/509] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:21.418860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.446339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.447092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.447367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.447635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.461523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.463134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:21.478761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:21.479780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:21.480580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:21.480580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.480580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:21.480580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 268/510] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:21.480839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.507542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.546682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bbf780-6694-4c98-86a0-8fd0867bf1b9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.584771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.585863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.586438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.586698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.598330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.600104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.611297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7796a24e-d1c5-44b6-823d-c5e181bec48f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:21.617555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:21.618903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:21.620261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:21.620261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.620261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:21.620261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 243/511] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:21.620705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.654577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.655348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.655597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.655843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.667310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.668826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:21.684923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:21.686086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:21.687078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:21.687078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.687078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:21.687078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 269/512] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:21.687605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.701324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.749622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f2e4bd-5ddd-4479-8bee-dc81c8410d6d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.768012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.792994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.793252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.793394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.793663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.794617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:21.795122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.797393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.812693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 244/513] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:21.813536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.823023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f893f48c-57f8-41c8-892a-7e410e6b267d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.868610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.869237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.869515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.869785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.887533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.889010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:21.905856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:21.907043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:21.908791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:21.908791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:21.908791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:21.911604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 270/514] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:21.912447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.933476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b5b378-fa2d-46e1-b9dc-3981f7cfee0f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.972365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.973458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.973746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.974016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.975807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:21.976125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.977512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.982677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:21.996966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 245/515] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:21.999816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.026014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ac9b60-8ee4-4625-8672-e7d244c85c2b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.036867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:22.061853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.062482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.062758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.063024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.063656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:22.063988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.065484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.076977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 271/516] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:22.077764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.099817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb682fe-23d0-486a-9f37-3af7f24062c9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.132950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.133556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.133833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.134102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.134775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:22.135074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.136425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.148336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 246/517] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:22.149128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.177505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c5fb0-0ff2-458d-95d8-819a13153b61 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.195198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:22.212397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.213031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.213309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.213584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.221096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.222558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:22.234478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:22.235416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 272/518] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/users/cinder => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:22.236761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.255508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:22.259164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4181edf2-67cd-486b-aa63-7990ae8a751c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.293751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.294356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.294659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.294928 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.295569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:22.295870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.297236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.308283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 247/519] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:22.309073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.336747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec328cc4-cb90-4001-8de6-8c669fb677f8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.371037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.371646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.371933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.372200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.372836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.374283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.412226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8379a75c-7bae-4e20-a7a5-8cb6b963f3dd None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.444286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.444926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.445199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.445465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.451848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.453275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:22.465427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:22.466334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:22.470140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:22.470618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 273/520] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:33:22 2022] POST /identity/v3/users => generated 247 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:22.470618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.470747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 248/521] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:22.470747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.492186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:22.508448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:22.583975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7807b4c0-59e9-497c-b5df-73425be074ed None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.585234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01e4ca-a0b6-4f7b-ad7e-57dc3577f6fc None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.621930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.622603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.623234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.623522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.624280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:22.624589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.626723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.628763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.629528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.629905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.637863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.639366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.647432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 249/522] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:22.648354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:22.654743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:22.655826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:22.656854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:22.657285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 274/523] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:22.657974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.747898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:22.766587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:22.771207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe037a-df80-4396-8b2e-19da01ba99a1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.773939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1b066c-6dc3-4789-a324-35c4f00379fd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.808470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.808710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.809297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.809726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.809818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.810454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.810553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.810832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.811416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:22.811727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.813413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.820190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:22.821703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:22.833510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:22.834766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:33:22.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:22.845883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 250/524] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/users/neutron => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:22.845993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 275/525] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:22.846081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.846162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:22.958846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d787c404-587e-493a-b108-d8274b8cceb8 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.961719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0237b252-a934-44c5-8ee6-9de5659022aa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:22.994746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.995348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.995620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.995892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.997629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:22.998239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:22.998543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:22.998812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:22.999495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:22.999789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.001044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:23.001235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.014537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 276/526] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:23.015842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.032796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:23.084453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.086802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.120763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7091044b-92ab-42dd-9b43-98c6ee544fc4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:23.157537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:23.158475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:23.158747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:23.159016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:23.159743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:23.160036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.161468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.179936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 277/527] 10.222.0.27 () {56 vars in 1139 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:23.180979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.201269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 251/528] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:23.201269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.239390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:23.266675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:23.297362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4601a2b-3bbb-4124-a9e5-56c27b7ba459 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:23.310891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ef5174-a0e7-431a-84c2-dbe5db67f26b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:23.333259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:23.333998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:23.334373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:23.334764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:23.335593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=40e12aa547b84ac38fd222a9f5f62ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:23.336012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.337551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.345650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 278/529] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/role_assignments?user.id=40e12aa547b84ac38fd222a9f5f62ae2&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:23.346384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.357803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:23.358766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:23.359190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:23.360308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:23.361896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=146238397f514eada174ab3374be5418) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:23.362945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.365164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.375586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 252/530] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/role_assignments?user.id=146238397f514eada174ab3374be5418&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:23.376780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.469121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089600b-65bd-44fe-ba07-4014a320cde6 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:23.508297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:23.509100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:23.509476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:23.509844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:23.514761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:23.536478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:23.541826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b79e402-8065-458b-af08-c6a33ec2c5a9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:23.586681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.588126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.599230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:23.599926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:23.600155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:23.600425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:23.623392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.624899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:23.652690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:23.653820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:23.654666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:23.654666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:23.654666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:23.654666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 253/531] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:23.661316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.797015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 279/532] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:23.803609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.809367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0a31cd-476d-46bc-9c0b-704d29563b70 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:23.865167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:23.866068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:23.866353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:23.866655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:23.867300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:23.867599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:23.869220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:23.882419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:23.887007 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 254/533] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:23.893063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:23.922794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:24.026752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454c07fc-0e75-4a65-b791-b6501520973d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:24.037686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668629dd-9597-45fa-9c1f-a694e1f7f530 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:24.084604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:24.085248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.085522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.085785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.100994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:24.102460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:24.104082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:24.104723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.104991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.105254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:24.120042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:24.120891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:24.121699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 255/534] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:24.127657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:24.131348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:24.132834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:24.172505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:24.173539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:24.174513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:24.174513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:24.174513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:24.174513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 280/535] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:24.180514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:24.238865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:24.301312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c14aeb-ab3a-4fac-ad3b-72b0fa14d19c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:24.371299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:24.375689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:24.376968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.377233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.377436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.378106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:24.378434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:24.380714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:24.400811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bb2149-e438-4e5c-b37a-a2c1f655b363 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:24.421740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 256/536] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:24.434431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:24.465230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:24.465848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.466127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.466397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.467063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:24.467365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:24.468747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:24.504080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 281/537] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:24.509460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:24.598432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a73356-86c6-47ae-8982-1a296116b828 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:24.629706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:24.635397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:24.636259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.636259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.636525 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.637256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=f49a19f9484d4aa99567e66b4fc396ef) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:24.637562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:24.639028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:24.645136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 257/538] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:24 2022] GET /identity/v3/role_assignments?user.id=f49a19f9484d4aa99567e66b4fc396ef&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 259 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:24.682614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac161fea-7978-4e33-b93b-5e012044ae1d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:24.730386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e474e1c1-33c6-461d-a6b7-0d070149bcaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:24.731352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e474e1c1-33c6-461d-a6b7-0d070149bcaf None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.731789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e474e1c1-33c6-461d-a6b7-0d070149bcaf None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.732213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e474e1c1-33c6-461d-a6b7-0d070149bcaf None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.746593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e474e1c1-33c6-461d-a6b7-0d070149bcaf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:24.748411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e474e1c1-33c6-461d-a6b7-0d070149bcaf None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-e474e1c1-33c6-461d-a6b7-0d070149bcaf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:24.772315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:24.773457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:24.774540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:24.780711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 282/539] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:24.789663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:24.907346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35858db3-d418-42d5-a552-6a54c828b3a1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:24.940740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da99e244-0dce-45f1-83ac-c1c66728ae50 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.941399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da99e244-0dce-45f1-83ac-c1c66728ae50 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.941399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da99e244-0dce-45f1-83ac-c1c66728ae50 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.943640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 283/540] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:24.949210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4458f82d-ada5-4432-9164-cf47935deaf3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.949760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4458f82d-ada5-4432-9164-cf47935deaf3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.950032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4458f82d-ada5-4432-9164-cf47935deaf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.959256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:24.959741 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:24.960019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:24.960287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:24.960987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:24.966287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:24.967492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:24.980878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-4458f82d-ada5-4432-9164-cf47935deaf3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:24.982158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4458f82d-ada5-4432-9164-cf47935deaf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:24.984561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:24.990854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 258/541] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:24.997424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:25.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:25.113583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 284/542] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:25.141234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e16953-71ec-4088-b877-5d5113c0c2ac None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:25.176845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:25.177376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:25.177655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:25.177927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:25.178678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=df391d2bf7884538823f04db5564d02c) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:25.178989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:25.180421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:25.186281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 259/543] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:24 2022] GET /identity/v3/role_assignments?user.id=df391d2bf7884538823f04db5564d02c&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:25.472565 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4458f82d-ada5-4432-9164-cf47935deaf3 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:25.609413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4458f82d-ada5-4432-9164-cf47935deaf3 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:25.650255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:25.650924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:25.651207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:25.651480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:25.659188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:25.663789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:25.668970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:25.669863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:25.670686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 285/544] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:25.678033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:25.788187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62da10b7-8137-49a4-b625-fd8111b11954 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:25.810766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:25.828087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:25.828966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:25.830173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:25.830688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:25.837355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:25.837355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:25.837355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:25.842447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 260/545] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:25.848642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:25.993354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc30a47-2457-4167-bcc8-8a433c6ae926 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.055974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9c1f47-de9c-4c2a-8502-f831677a9319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.056552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9c1f47-de9c-4c2a-8502-f831677a9319 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.056826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9c1f47-de9c-4c2a-8502-f831677a9319 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.057091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9c1f47-de9c-4c2a-8502-f831677a9319 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.072164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9c1f47-de9c-4c2a-8502-f831677a9319 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:26.073627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9c1f47-de9c-4c2a-8502-f831677a9319 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-ed9c1f47-de9c-4c2a-8502-f831677a9319 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:26.092389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:26.093403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:26.096100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:26.096100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:26.096100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:26.096100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 286/546] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:26.099907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.111984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c98222-f36c-458b-b9c8-abcf2c3e2450 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.112264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c98222-f36c-458b-b9c8-abcf2c3e2450 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.112539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c98222-f36c-458b-b9c8-abcf2c3e2450 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.114190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 287/547] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:26.119403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954ab68-7027-4c66-a0b3-8c6b381b6d00 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.119686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954ab68-7027-4c66-a0b3-8c6b381b6d00 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.119961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954ab68-7027-4c66-a0b3-8c6b381b6d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.172646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a954ab68-7027-4c66-a0b3-8c6b381b6d00 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:26.174054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a954ab68-7027-4c66-a0b3-8c6b381b6d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:26.220270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cdd975-4046-41f3-b0c7-2c51afa89785 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.224793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:26.247275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 288/548] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:26.249679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da878c2-e067-4e90-979f-ba5eef5f595d None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.249679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da878c2-e067-4e90-979f-ba5eef5f595d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.250079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da878c2-e067-4e90-979f-ba5eef5f595d None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.252454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 289/549] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:26.258101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d072a-da0c-443b-bf2a-b4247b41ee6d None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.258485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d072a-da0c-443b-bf2a-b4247b41ee6d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.258832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d072a-da0c-443b-bf2a-b4247b41ee6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.260152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.260977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.261419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.261842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.262674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:26.263119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:26.264620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:26.279264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 261/550] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:26.286198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.298251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ab2d072a-da0c-443b-bf2a-b4247b41ee6d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:26.299725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ab2d072a-da0c-443b-bf2a-b4247b41ee6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:26.380990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:26.434330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 290/551] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:26.462529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05c10a8-584e-4fd1-8490-8b509c011a0a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.500351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.501033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.505651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.505651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] PATH_INFO: `/v3/users/swift` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.510283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:26.511871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:26.524274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:26.525264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:26.526255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 262/552] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/users/swift => generated 83 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:26.532345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d072a-da0c-443b-bf2a-b4247b41ee6d None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.560376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.701553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ba8cc8-10e0-4082-b9da-21dbc648d34e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.703165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:26.717106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d072a-da0c-443b-bf2a-b4247b41ee6d None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.743682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:26.746449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.747154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.747433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.747703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.768594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:26.776902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:26.777798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:26.778650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 263/553] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:26.787306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.804130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.805177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.805673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.806153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.807134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:26.807641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:26.809287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:26.828834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 291/554] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:26.830806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.908460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e25a325-6070-4852-91a2-5fef26db26f2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.949636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ec7d8a-04ba-4c06-bc4e-52543b54661c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.950994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.951836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.952130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.952476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.953845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:26.953845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:26.955340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:26.969132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 264/555] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:26.970999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.994962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.996422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.996891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.997585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.998536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.012800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:27.019572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:27.020539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:27.021348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:27.023793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 292/556] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:27.025037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.061805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.087129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5113d45f-dcea-4bf6-8ebe-f4ad0464ebe1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.133999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.135161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.135746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.136209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.139504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48a7d78-0fcf-4abc-be16-93e2d27ece50 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.190922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.192492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.193120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.193643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.209652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.211640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.216727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.218509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:27.231837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:27.233045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:27.234082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:27.234082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:27.234082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:27.237686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 293/557] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:27.239020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.255320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.330345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.349695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486774d-8238-42d5-9fcb-31089ca3b542 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.353502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 265/558] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:27.354494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.397166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e014d8-661a-4d94-8ded-7dd78f63010d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.397788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e014d8-661a-4d94-8ded-7dd78f63010d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.398068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e014d8-661a-4d94-8ded-7dd78f63010d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.398340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e014d8-661a-4d94-8ded-7dd78f63010d None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.399082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e014d8-661a-4d94-8ded-7dd78f63010d None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:27.399386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e014d8-661a-4d94-8ded-7dd78f63010d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.400777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e014d8-661a-4d94-8ded-7dd78f63010d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.410434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 294/559] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:27.411694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ce834-6173-48e4-b273-42ae8e2a6235 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.411979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ce834-6173-48e4-b273-42ae8e2a6235 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.412248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ce834-6173-48e4-b273-42ae8e2a6235 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.417796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 295/560] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:27 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:27.417796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001ce834-6173-48e4-b273-42ae8e2a6235 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.490054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a450bc5-8813-42bb-abd3-765fd54ae070 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.516350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.527884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001ce834-6173-48e4-b273-42ae8e2a6235 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.535607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.535607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.535607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.535607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.535989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:27.535989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.535989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.549255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 266/561] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:27.549917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.566187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2354ab66-dbd1-4be9-b403-c67978f73699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.566966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2354ab66-dbd1-4be9-b403-c67978f73699 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.567312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2354ab66-dbd1-4be9-b403-c67978f73699 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.567765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2354ab66-dbd1-4be9-b403-c67978f73699 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.578315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2354ab66-dbd1-4be9-b403-c67978f73699 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.580966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2354ab66-dbd1-4be9-b403-c67978f73699 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.582816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-2354ab66-dbd1-4be9-b403-c67978f73699 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:27.596701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:27.597667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:27.598544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:27.598544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:27.598544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:27.602456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 296/562] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:27.603979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16db856e-83b7-4b1e-b9ff-bc83b5449b71 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.603979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16db856e-83b7-4b1e-b9ff-bc83b5449b71 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.604540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16db856e-83b7-4b1e-b9ff-bc83b5449b71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.695575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-16db856e-83b7-4b1e-b9ff-bc83b5449b71 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:27.696785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-16db856e-83b7-4b1e-b9ff-bc83b5449b71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:27.721671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dfb70d-92bd-4176-a35b-4f0d90f890da None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.762843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.763394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.763672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.763942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.774868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 297/563] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:27.775133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.776081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16db856e-83b7-4b1e-b9ff-bc83b5449b71 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.776601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:27.791542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:27.792538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:27.793405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:27.793405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:27.793405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:27.793405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 267/564] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:27.793767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.839803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.900027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16db856e-83b7-4b1e-b9ff-bc83b5449b71 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.912320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e53dd24-1c4a-4034-b05f-b950d3b98ab6 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:27.925209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:27.936008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.936715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.936975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.937250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] PATH_INFO: `/v3/users/glance` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.944960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.946395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.948410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e2dbd0-d7c1-4afe-ab56-117497ee8497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:27.949193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e2dbd0-d7c1-4afe-ab56-117497ee8497 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.949623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e2dbd0-d7c1-4afe-ab56-117497ee8497 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.950032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e2dbd0-d7c1-4afe-ab56-117497ee8497 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.950836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e2dbd0-d7c1-4afe-ab56-117497ee8497 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:27.951300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e2dbd0-d7c1-4afe-ab56-117497ee8497 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:27.952811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e2dbd0-d7c1-4afe-ab56-117497ee8497 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:27.958843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:27.959948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:33:27.961041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:27.964484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 298/565] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/users/glance => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:27.965347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:27.969129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 268/566] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:27.970556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf9d0da-ee9c-4c82-ac38-93c9d4eb7ac3 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:27.971031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf9d0da-ee9c-4c82-ac38-93c9d4eb7ac3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:27.971446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf9d0da-ee9c-4c82-ac38-93c9d4eb7ac3 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:27.973148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 269/567] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:27.974318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf9d0da-ee9c-4c82-ac38-93c9d4eb7ac3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.075901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.076888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b5b6cf-9168-437d-8c12-fd0e02da57eb None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.083158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf9d0da-ee9c-4c82-ac38-93c9d4eb7ac3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.110821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.111427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.111703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.111977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.112612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:28.112913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.114298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.117264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45766147-3947-4633-bfce-65298515e8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.117864 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45766147-3947-4633-bfce-65298515e8f3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.118393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45766147-3947-4633-bfce-65298515e8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.118691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45766147-3947-4633-bfce-65298515e8f3 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.119426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45766147-3947-4633-bfce-65298515e8f3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:28.119725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45766147-3947-4633-bfce-65298515e8f3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.121127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45766147-3947-4633-bfce-65298515e8f3 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.126109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 299/568] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:28.127190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.146810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 270/569] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Oct 4 08:33:27 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:28.146810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed008731-16d2-4f46-8ee8-74e7922ce3b5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.146810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed008731-16d2-4f46-8ee8-74e7922ce3b5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.146810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed008731-16d2-4f46-8ee8-74e7922ce3b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.161550 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ed008731-16d2-4f46-8ee8-74e7922ce3b5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:28.162802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ed008731-16d2-4f46-8ee8-74e7922ce3b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:28.177263 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.228775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 271/570] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:28.229601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed008731-16d2-4f46-8ee8-74e7922ce3b5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.229771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5cc73-af1a-43f0-9755-ec0a32e1cd62 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.267683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.268319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.268600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.268870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.276569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.278005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:28.289660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:28.290639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 300/571] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:28.291504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.328490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.347524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed008731-16d2-4f46-8ee8-74e7922ce3b5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.395697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.396148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae17a40-33b9-4668-8c1c-39cb12a06cd3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.396348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.396609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.396876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.398476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.408398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:28.413158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:28.414070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:28.414918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 272/572] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:28.415703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.437677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.438308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.439864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.440079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.447224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.448723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:28.461585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:28.467663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 301/573] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:28.468826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.523669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4e9c9d-c201-4371-907e-95031dea6da2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.557530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.558148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.558449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.558724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.568191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc552995-0847-4c01-90d2-7c59b478e1f5 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.568514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.569948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.572426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:28.590461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:28.591456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:28.592306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:28.592306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:28.592306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:28.592306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 273/574] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:28.592306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.602131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.602771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.603051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.603317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.604026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:28.604326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.605740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.618645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 302/575] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:28.622233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.643340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.692970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b923783-97e3-4a57-8a4d-21322f940c8b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.714810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a2550e-d7aa-47f1-9192-f44376e68207 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.725949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.726586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.726865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.727138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.727797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:28.728098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.729464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.737649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 274/576] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:28.738462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.754314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.754955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.755237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.755507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.756232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:28.756532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.757949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.775587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 303/577] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:28.776385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.799242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.841237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b504ae-d2d4-4f63-b1d6-4af284416c0a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.856021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:28.873647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a7489-2f40-4f6e-b082-f4ff1fe4ecef None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:28.874652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.875274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.875545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.875816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.885783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:28.891344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:28.892929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:28.893905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 275/578] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:28.894657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.913401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:28.914005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.914283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.914586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.915226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:28.915531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:28.916902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:28.928604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 304/579] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:28.931294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:28.990187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc6593-37e0-4f2c-a470-76276c3124ed None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.007578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.024297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.024899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.025176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.026189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.026189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=085a0cc3f5cb477989e6e75adea7c53a) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:29.026544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.027937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.033925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 276/580] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/role_assignments?user.id=085a0cc3f5cb477989e6e75adea7c53a&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:29.035958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65ca92b-c766-40c8-853b-b2ba387240f3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.038686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.083892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.084648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.084990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.085271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.098287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.099958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.102792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:29.117307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:29.120944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:29.121817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:29.121817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:29.121817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:29.121817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 305/581] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:29.121817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.161188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7fab09-5109-4705-abb0-aac54e0a96b7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.200698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.201342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.201617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.201884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.215856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.217299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.231895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d427109-dd58-4f7b-bcc8-b97ff117c03e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:29.232068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:29.232948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:29.233812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:29.233812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:29.233812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:29.237775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 277/582] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:29.238681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.275275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.277143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.277762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.278037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.278304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.278990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:29.279286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.280645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.288974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 306/583] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:29.289694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.350307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a250912c-db90-482f-a247-f06d88fc4455 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.351402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.392046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.393114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.393114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.393384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.401189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.402757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.404875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89ccc8c-0d08-4d84-a7d6-c122cf0c7ca8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:29.415304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:29.416202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:29.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 278/584] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:29.417556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.444510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.445118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.445398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.445671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.446310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:29.446645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.448009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.464475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 307/585] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:29.464475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.527126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.535788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051650ae-5cb6-42d8-a116-5371940a2b20 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.564927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f19fa-f376-4288-bdfb-6b4cb921470c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.575145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.575867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.576141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.576416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.577057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:29.577360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.580178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.582216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.597724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 279/586] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:29.598654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.602179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.602799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.603082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.603357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.614666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.616311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:29.634770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:29.635697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:29.636553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:29.636553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:29.636553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:29.636553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 308/587] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:29.637058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.704672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3871e-302a-447e-93e5-2ae746fdcae5 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.739915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.740557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.740782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.741051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.741736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:29.742048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.743749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.748305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbed658-3da3-49ff-a8e3-3711b6d67aed None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.757111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 280/588] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:29.757925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.777667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.790883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.791802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.792202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.792475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.800633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.802089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:29.814286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:29.815241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 309/589] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:29.816250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.844585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:29.873490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3460f6f6-0141-4bd3-ad57-92d22e9a5023 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.911119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.911766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.912044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.912316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.918324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560efe80-db12-4a23-853a-c08af70ef1bd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.919587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.921037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:29.932946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:29.933878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 281/590] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:29.935202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.960078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.960679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.960955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.961222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.961852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:29.962148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.963533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.979817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 310/591] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:29.979817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.026179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:30.053966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e95d47b-ed9a-45ed-b7a1-48cc3eaade87 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.066629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:30.078598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1b0340-1747-4c60-a725-e61df1197814 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.092678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.093330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.093605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.093872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/146238397f514eada174ab3374be5418/roles/4b9e5859f89942e79cfb0e553769100b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.110224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=146238397f514eada174ab3374be5418, role_id=4b9e5859f89942e79cfb0e553769100b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.112444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.115896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.116508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.116780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.117050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.117770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:30.118071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.119777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.138138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 311/592] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:30.140199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.183615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 282/593] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:29 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/146238397f514eada174ab3374be5418/roles/4b9e5859f89942e79cfb0e553769100b => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:30.193468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.259762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ee30c4-5057-47c5-98ee-2ad439e0cd67 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.302078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.302743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.303046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.304112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.304849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:30.305378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.306888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.320948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 312/594] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:30.322783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.341428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:30.361652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c7bfe7-eda7-4b68-b083-a7fd7a7e2065 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.395318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:30.409461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.410502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.410986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.411471 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.422191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.424644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:30.440775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:30.441758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:30.442948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 283/595] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:30 2022] GET /identity/v3/users/neutron => generated 85 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:30.443540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.469008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac7fd58-4389-48d0-adad-4e74570a1f49 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.508780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.509797 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.513300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.513300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.530161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.533494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:30.558679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:30.559812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:30.560724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:30.560724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:30.560724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:30.561427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 313/596] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:30.562552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.589072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3db4f-0779-4fa8-b22b-a7e697d24249 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.638258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.639102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.639477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.639848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/40e12aa547b84ac38fd222a9f5f62ae2/roles/4b9e5859f89942e79cfb0e553769100b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.646704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:30.657981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=40e12aa547b84ac38fd222a9f5f62ae2, role_id=4b9e5859f89942e79cfb0e553769100b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.660747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.691359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aa7446-5a1f-414f-825a-b5e36d7f2bcb None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.726876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.727774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.728228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.728643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.729568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:30.730011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.731686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.753126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 314/597] 10.222.0.27 () {56 vars in 1139 bytes} [Tue Oct 4 08:33:30 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:30.753760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.761065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 284/598] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:30 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/40e12aa547b84ac38fd222a9f5f62ae2/roles/4b9e5859f89942e79cfb0e553769100b => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:30.762055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:30.806765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:30.886163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f13cdeb-f625-4a46-a393-89b038676827 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.942096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1979fbc-6bd0-4d95-8b9d-a075ab666991 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:30.953050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:30.953709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:30.953986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:30.954870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:30.977275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:30.978760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:30.980155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:30.999578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:31.000568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:31.002392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:31.002392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:31.002392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:31.002392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 285/599] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:31.008673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:31.014032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.014724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.015071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.015288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.015940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:31.016246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.018066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.033848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 315/600] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:31.041210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:31.116860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:31.134509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e1bd4f-4bf5-4d2a-8007-769a4b0ae95a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:31.189502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dd2654-9f98-4c94-89b2-5bf0c1fb3679 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:31.196650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.197537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.198020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.198529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.199382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:31.199895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.201522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 286/601] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:31.233374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:31.269496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.270545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.271045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.271528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.287143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.288905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:31.307893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:31.308844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:31.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:31.316769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 316/602] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:31.321735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:31.359574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a3cd89-5b70-42a5-bc3a-a2b52b009b45 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:31.371077 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:31.400402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80b3ef9-761d-4a5c-addc-1f947abfbcd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.401019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80b3ef9-761d-4a5c-addc-1f947abfbcd6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.401296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80b3ef9-761d-4a5c-addc-1f947abfbcd6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.401610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80b3ef9-761d-4a5c-addc-1f947abfbcd6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.410892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80b3ef9-761d-4a5c-addc-1f947abfbcd6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.412388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80b3ef9-761d-4a5c-addc-1f947abfbcd6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-c80b3ef9-761d-4a5c-addc-1f947abfbcd6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:31.424683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:31.425581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:31.426968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 287/603] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:31.427442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b474ab1e-aff6-40ab-9d26-5c155806263d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.427716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b474ab1e-aff6-40ab-9d26-5c155806263d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.427864 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b474ab1e-aff6-40ab-9d26-5c155806263d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.433334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 288/604] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:31.434164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b474ab1e-aff6-40ab-9d26-5c155806263d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:31.444247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:31.448748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d993f9f-d97b-436d-926b-adbbd4dde021 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:31.485197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16acb350-aa3a-4659-801d-72a941c4994b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.486644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16acb350-aa3a-4659-801d-72a941c4994b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.487069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16acb350-aa3a-4659-801d-72a941c4994b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.487597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16acb350-aa3a-4659-801d-72a941c4994b None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.489597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16acb350-aa3a-4659-801d-72a941c4994b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:31.490051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16acb350-aa3a-4659-801d-72a941c4994b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.491661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16acb350-aa3a-4659-801d-72a941c4994b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.505557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 317/605] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:31.507175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126720f9-a922-4a77-ab9b-102cba1b0883 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.507604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126720f9-a922-4a77-ab9b-102cba1b0883 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.508045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126720f9-a922-4a77-ab9b-102cba1b0883 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.527628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-126720f9-a922-4a77-ab9b-102cba1b0883 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:31.529129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-126720f9-a922-4a77-ab9b-102cba1b0883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:31.542172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b474ab1e-aff6-40ab-9d26-5c155806263d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:31.577640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.577932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.578213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.578507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.579274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:31.579584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.581002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.593522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 289/606] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:31.594332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:31.614435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 318/607] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:31.615746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126720f9-a922-4a77-ab9b-102cba1b0883 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:31.661344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:31.668574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:31.699133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d1588f-18b3-4e2f-a4b5-304668bc83fa None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:31.742429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.742429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.742429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.742429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/f49a19f9484d4aa99567e66b4fc396ef/roles/4b9e5859f89942e79cfb0e553769100b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.752940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=f49a19f9484d4aa99567e66b4fc396ef, role_id=4b9e5859f89942e79cfb0e553769100b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.759657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.784875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126720f9-a922-4a77-ab9b-102cba1b0883 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:31.823213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:31.823885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:31.824163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:31.824432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/df391d2bf7884538823f04db5564d02c/roles/4b9e5859f89942e79cfb0e553769100b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:31.832890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 290/608] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:31 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/f49a19f9484d4aa99567e66b4fc396ef/roles/4b9e5859f89942e79cfb0e553769100b => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:31.847917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=df391d2bf7884538823f04db5564d02c, role_id=4b9e5859f89942e79cfb0e553769100b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:31.849393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:31.966949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 319/609] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:31 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/df391d2bf7884538823f04db5564d02c/roles/4b9e5859f89942e79cfb0e553769100b => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:32.013807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:32.140050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634e7798-8703-4030-8f6c-8daaf3389e93 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:32.161336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:32.182150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:32.183174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:32.183680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:32.184148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:32.196184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:32.201407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:32.202308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:32.203348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:32.205985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 291/610] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:32.212146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:32.295714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:32.320019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae68af0-82ff-4ecb-952c-67bb506840ff None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:32.356695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:32.357369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:32.357648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:32.357920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:32.358776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:32.359082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:32.360501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:32.375309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:32.376740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 320/611] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:32.520900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831e374e-4266-4dad-985b-460f7abd280c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:32.561896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:32.562469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:32.562751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:32.563020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:32.574590 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:32.576047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:32.599907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:32.600892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:32.601762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:32.601762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:32.601762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:32.603892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 292/612] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:32.609372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:32.748324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694e2c1-365c-4c4c-b1a3-26a0d9c00977 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:32.799165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ee6b42-5d0e-47d1-a276-a37f24c46ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:32.800211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee6b42-5d0e-47d1-a276-a37f24c46ded None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:32.800703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee6b42-5d0e-47d1-a276-a37f24c46ded None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:32.801162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee6b42-5d0e-47d1-a276-a37f24c46ded None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:32.802027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ee6b42-5d0e-47d1-a276-a37f24c46ded None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:32.802550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ee6b42-5d0e-47d1-a276-a37f24c46ded None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:32.804156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ee6b42-5d0e-47d1-a276-a37f24c46ded None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:32.815911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:32.819277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 321/613] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:32.827535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:32.869290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:32.939336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480023b-c7cd-437f-b7d1-c414d8555f50 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:32.939647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480023b-c7cd-437f-b7d1-c414d8555f50 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:32.939869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480023b-c7cd-437f-b7d1-c414d8555f50 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:32.941200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 322/614] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:32.946614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b87046b-c3d0-458e-9534-a84914fd10fb None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:32.946948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b87046b-c3d0-458e-9534-a84914fd10fb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:32.947228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b87046b-c3d0-458e-9534-a84914fd10fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:32.977616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6b87046b-c3d0-458e-9534-a84914fd10fb None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:32.978845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6b87046b-c3d0-458e-9534-a84914fd10fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:32.990099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8658c-4d06-4494-a07c-d452e3c4f417 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:33.070145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:33.070739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.071016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.088936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:33.090399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:33.117908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:33.119226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:33.120123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:33.120636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 293/615] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:33.127169 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:33.132906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 323/616] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:33.238630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaea3f1-c6b1-4890-9711-f7da16483e5f None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.239385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaea3f1-c6b1-4890-9711-f7da16483e5f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.239385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaea3f1-c6b1-4890-9711-f7da16483e5f None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.241061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:33.241243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 324/617] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:33.247477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef76bf0-b250-4894-bf6c-902654aa7ae8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.247761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef76bf0-b250-4894-bf6c-902654aa7ae8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.248376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef76bf0-b250-4894-bf6c-902654aa7ae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.254238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f307004b-5e4d-4967-baf6-c836f9951ee0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:33.293181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6ef76bf0-b250-4894-bf6c-902654aa7ae8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:33.294380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6ef76bf0-b250-4894-bf6c-902654aa7ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:33.316892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:33.317553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.317834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.318105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.319009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:33.319533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:33.321004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:33.347438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 294/618] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:33.353125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:33.441144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 325/619] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:33.471541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba70d67-4cbc-477e-a743-30cd467d3cfb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:33.508006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:33.508624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.508895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.509159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.520106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:33.521549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:33.528495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:33.537497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:33.539984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:33.542137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:33.542137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:33.542137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:33.542137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 295/620] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:33.544332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef76bf0-b250-4894-bf6c-902654aa7ae8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:33.633269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:33.685107 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:33.691770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef76bf0-b250-4894-bf6c-902654aa7ae8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:33.765132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:33.765765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.766046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.766314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.767208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:33.767528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:33.770078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:33.784563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 326/621] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:33.785390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:33.863676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497eaf3d-5a54-411f-a423-e965275bbabe None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:33.905701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:33.906447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.906658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.906817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.915243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd54738-c8b9-49f4-b826-515c7f3d72c4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:33.918421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:33.924577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:33.926719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:33.928657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:33.938438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 296/622] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:33.938438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:33.938910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:33.952398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:33.953323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.953612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.953889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.968901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:33.976488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:33.978671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 327/623] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:33.980730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.002550 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:34.049306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a816d1-ebf1-4371-a17d-615b0d4d5b74 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.093370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.094136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.094431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.094759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.102147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.103814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:34.116981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:34.119312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:34.121377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:34.130431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 297/624] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:34.130431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.130835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e638e10-4652-410a-88b2-9d933d35bba6 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.167611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.168298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.168583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.168856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.169521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:34.169823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.171266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.185858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 328/625] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:34.186802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.232562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:34.254890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f936f89a-f382-405e-936c-ca2e4767ac5d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.276810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:34.300231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.300857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.301141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.301419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.302760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:34.303075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.304789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.305773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416e8456-66c0-4c2b-aa5f-aac655256eaa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.317453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 298/626] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:34.318310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.362541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.363183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.363473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.364180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.364975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:34.365403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.367432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.384833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 329/627] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:34.385793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.446483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feb0709-5631-460e-94ea-c2284490cd49 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.483952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.485052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.486076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.486704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.501813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.503923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:34.520975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:34.521950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:34.522947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:34.522947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:34.522947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:34.529841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 299/628] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:34.530843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2536f6d-4681-4d36-82f5-89ce8645f52a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.531190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.533756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:34.565305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:34.570471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e07e22-58e8-4499-a471-6eba89857504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.571118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e07e22-58e8-4499-a471-6eba89857504 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.571399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e07e22-58e8-4499-a471-6eba89857504 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.571671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e07e22-58e8-4499-a471-6eba89857504 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.583544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e07e22-58e8-4499-a471-6eba89857504 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.585058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e07e22-58e8-4499-a471-6eba89857504 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-d1e07e22-58e8-4499-a471-6eba89857504 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:34.601232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:34.602208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:34.603093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:34.603093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:34.603093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:34.603093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 330/629] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:34.603853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80129a9b-cf19-4353-a0e8-07ee19e9830f None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.604128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80129a9b-cf19-4353-a0e8-07ee19e9830f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.604411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80129a9b-cf19-4353-a0e8-07ee19e9830f None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.609571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 331/630] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:34.609571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80129a9b-cf19-4353-a0e8-07ee19e9830f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.643673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f727b9-f482-476e-80fd-dabb0563f6fd None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.678260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640fd038-8f0b-4b82-a27e-f72a021f6cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.678944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640fd038-8f0b-4b82-a27e-f72a021f6cf7 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.679226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640fd038-8f0b-4b82-a27e-f72a021f6cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.679498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640fd038-8f0b-4b82-a27e-f72a021f6cf7 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/085a0cc3f5cb477989e6e75adea7c53a/roles/4b9e5859f89942e79cfb0e553769100b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.694006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640fd038-8f0b-4b82-a27e-f72a021f6cf7 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=085a0cc3f5cb477989e6e75adea7c53a, role_id=4b9e5859f89942e79cfb0e553769100b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.695453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640fd038-8f0b-4b82-a27e-f72a021f6cf7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.711466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80129a9b-cf19-4353-a0e8-07ee19e9830f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.744833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.745425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.745701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.745969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.746671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:34.746974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.748322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.759889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 332/631] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:34.760682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.765587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 300/632] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:34 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/085a0cc3f5cb477989e6e75adea7c53a/roles/4b9e5859f89942e79cfb0e553769100b => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:34.767503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3c7e1-cccc-4f62-9631-9a467af2a2fd None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.767792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3c7e1-cccc-4f62-9631-9a467af2a2fd None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.768065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3c7e1-cccc-4f62-9631-9a467af2a2fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.777226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:34.788433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-96b3c7e1-cccc-4f62-9631-9a467af2a2fd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:34.789689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-96b3c7e1-cccc-4f62-9631-9a467af2a2fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:34.875067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 301/633] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:34.877301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfe90e-75fd-4a1a-b58e-f387c67befd4 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.877588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfe90e-75fd-4a1a-b58e-f387c67befd4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.877858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfe90e-75fd-4a1a-b58e-f387c67befd4 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.884290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 302/634] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:34.884290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cfe90e-75fd-4a1a-b58e-f387c67befd4 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:34.899815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:34.904592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a37115-fe6d-46ab-a743-359e27646f95 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:34.956198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce79e0ab-ba52-41b2-a7b2-1f400ec91d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:34.957079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce79e0ab-ba52-41b2-a7b2-1f400ec91d96 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.957861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce79e0ab-ba52-41b2-a7b2-1f400ec91d96 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.957861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce79e0ab-ba52-41b2-a7b2-1f400ec91d96 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:34.958970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce79e0ab-ba52-41b2-a7b2-1f400ec91d96 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:34.959530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce79e0ab-ba52-41b2-a7b2-1f400ec91d96 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:34.960935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce79e0ab-ba52-41b2-a7b2-1f400ec91d96 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:34.975263 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 333/635] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:34.976147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad5d3a5-00ee-47aa-9be0-3233a7f9b79a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:34.976428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad5d3a5-00ee-47aa-9be0-3233a7f9b79a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:34.976698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad5d3a5-00ee-47aa-9be0-3233a7f9b79a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.000705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-aad5d3a5-00ee-47aa-9be0-3233a7f9b79a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:35.001970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-aad5d3a5-00ee-47aa-9be0-3233a7f9b79a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:35.012838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cfe90e-75fd-4a1a-b58e-f387c67befd4 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.059874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.060884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.061430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.061896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] PATH_INFO: `/v3/users/glance` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.072048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:35.074037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:35.083385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 334/636] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:35.084381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad5d3a5-00ee-47aa-9be0-3233a7f9b79a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.087066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:35.091591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:35.092552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:33:35.093466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:35.095732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 303/637] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:34 2022] GET /identity/v3/users/glance => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:35.101390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.218913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad5d3a5-00ee-47aa-9be0-3233a7f9b79a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.227132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559c6e5-29b0-40f3-890c-c6ca51a470db None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.234866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.263956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.264824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.265272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.265691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] PATH_INFO: `/v3/users/nova` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.268604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.269503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.270003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.270507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.271437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:35.271938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:35.273040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:35.273899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:35.274489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:35.290539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:35.291515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:35.292432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:35.296402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 335/638] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:35.297254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.306861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 304/639] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:35.308122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.372991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.423557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3c57c6-c635-477a-8583-95de17a2d1ab None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.433776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dd835d-b3bc-48e5-9f98-a931bda6bf49 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.469182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.470702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.471009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.471284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] PATH_INFO: `/v3/roles/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.472044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.477817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.478280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.478596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.478865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.479583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:35.479878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:35.481383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:35.498783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:35.499601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:35.500564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 336/640] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:35.501831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.501896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 305/641] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:35.501896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.603105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bac7f2-b029-45a5-84cf-389d7a1fbf9c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.611673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9bb9f3-acdc-448c-86f0-84e8d7af6009 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.628303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.640823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.641425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.642428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.642602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.648941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.649572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.649844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.650110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.663033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:35.667810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:35.668758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:35.669571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 306/642] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:35.671116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.719443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:35.722065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.722186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:35.780306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b73d574-8789-4a21-a5db-ce56f3681cb3 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.817958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.818696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.818971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.819242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.819898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:35.820195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:35.821780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:35.828918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 337/643] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:35.828918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.837475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 307/644] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:35.837475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:35.855139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.936806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d20fd-6180-4562-9239-fa0240134266 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.947054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e696b37d-870c-4dd0-ace7-f647d49f60be None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:35.957885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:35.973112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.973712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.973984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.974245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.982527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:35.983141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:35.983415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:35.983685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:35.984463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:35.984904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:35.986852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.001701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 308/645] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:36.002611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.043628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.044978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.069626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.109771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4bbe17-94a2-45a4-8b6f-703a971f264b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.146989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.147616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.147890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.148163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.151541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 338/646] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:36.152554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.159008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.160459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:36.175093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:36.176097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:36.177028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:36.177028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.177028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:36.177028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 309/647] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:36.177028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.208200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.272645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40de6bbf-2440-460d-b3db-7596f82dbff8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.289239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b48af9a-16c3-47c3-b02d-eea8934b67e9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.295940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.308859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.309818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.310258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.310717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.321763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.323455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.323897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.324525 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.324805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.325080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.335055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.336477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:36.339802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:36.340900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:36.341952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:36.341952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.341952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:36.348126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 339/648] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:36.349123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:36.350822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:36.351885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:36.352738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:36.352738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.352738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:36.352738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 310/649] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:36.352962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.451336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.458023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810fd13-a1a3-4897-900c-13aad1267c28 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.475891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73fe35c-28f1-4f66-9865-abe48af11104 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.492097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.492774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.493056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.493576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.504047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.505549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.513358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.514347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.514802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.515081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.515723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:36.516026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.517901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:36.521860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:36.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:36.523656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:36.523656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:36.523656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:36.528222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 340/650] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:36.528975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.534628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 311/651] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:36.534628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.562239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.637448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee54100b-6667-44c7-b2a2-b1c1e4f5fa74 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.643427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbd8cc7-b7ae-474e-81dd-24f75c8d5464 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.671307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.671916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.672204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.672457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.673716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:36.674018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.675428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.680524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.681125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.681401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.681664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.682296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:36.682703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.684070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.686537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.693149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 341/652] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:36.693149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.699538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 312/653] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:36.700298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.790626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.806385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d606bf51-0910-4e49-bfb1-2ba58a4fc357 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.812206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350082cc-241b-4768-8194-f43e2abd9f15 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.847255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.847502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.847954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.848050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.848340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.848438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.848744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.848830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.849682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:36.850057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.852050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.858931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.862315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.870362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 342/654] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:36.870362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:36.877527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:36.878614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:36.879701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:36.880666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 313/655] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:36.881317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.946393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:36.986444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0988c4a-b549-4c35-8646-75de672e6ec0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.992794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea886908-def5-4651-8a78-496b835453a1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.020497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.021122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.021392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.021655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] PATH_INFO: `/v3/users/neutron` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.029215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.030604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.030752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.031231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.031504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.031771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.040696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.042273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:37.043475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:37.044544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:33:37.045652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:37.046543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 343/656] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/users/neutron => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:37.047376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:37.057304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:37.058475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:33:37.059592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:37.061080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.062549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 314/657] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:37.064016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.153443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad6cbf5-d308-4bf1-8deb-bd628b239dbd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.162539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12529353-8580-4079-96ef-bee2a8f76b32 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.178190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.186068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df24c8a-3a93-459c-a26a-88e6634c477f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.186712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df24c8a-3a93-459c-a26a-88e6634c477f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.186983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df24c8a-3a93-459c-a26a-88e6634c477f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.187250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df24c8a-3a93-459c-a26a-88e6634c477f None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.193749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df24c8a-3a93-459c-a26a-88e6634c477f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.198775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.198890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df24c8a-3a93-459c-a26a-88e6634c477f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.198975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.198975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.198975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.198975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:37.198975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.198975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0df24c8a-3a93-459c-a26a-88e6634c477f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:37.206632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:37.207473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:37.208561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 344/658] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:37.209429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e97c8cb-0525-4450-93b5-32f199445734 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.209429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e97c8cb-0525-4450-93b5-32f199445734 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.209674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e97c8cb-0525-4450-93b5-32f199445734 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.210902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 345/659] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:37.211582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e97c8cb-0525-4450-93b5-32f199445734 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.215956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 315/660] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:37.217011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.277803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.314022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e97c8cb-0525-4450-93b5-32f199445734 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.315809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cfbcaa-0242-4982-b438-31b58c3f7880 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.347537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.348135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.348414 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.348603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81018bc8-ab89-4d7a-b67a-ac6be909aee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.348710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.349195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81018bc8-ab89-4d7a-b67a-ac6be909aee0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.349380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:37.349469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81018bc8-ab89-4d7a-b67a-ac6be909aee0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.349682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.349772 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81018bc8-ab89-4d7a-b67a-ac6be909aee0 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.350425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81018bc8-ab89-4d7a-b67a-ac6be909aee0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:37.350733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81018bc8-ab89-4d7a-b67a-ac6be909aee0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.351148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.352123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81018bc8-ab89-4d7a-b67a-ac6be909aee0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.368588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 346/661] 10.222.0.27 () {56 vars in 1139 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:37.369404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.369992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 316/662] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:37.371274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cd29b-81b0-42c8-97cb-328c34a8bd04 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.371557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cd29b-81b0-42c8-97cb-328c34a8bd04 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.371826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cd29b-81b0-42c8-97cb-328c34a8bd04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.392339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6d8cd29b-81b0-42c8-97cb-328c34a8bd04 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:37.393592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6d8cd29b-81b0-42c8-97cb-328c34a8bd04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:37.421189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.463538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 317/663] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:37.464372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cd29b-81b0-42c8-97cb-328c34a8bd04 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.482275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fb18c6-a60b-4332-85e3-ee5288cc7a6d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.508795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.520234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.520846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.521123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.521392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.522193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:37.522520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.524168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.537028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 347/664] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:37.537828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.572184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cd29b-81b0-42c8-97cb-328c34a8bd04 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.606622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.607214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.607486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.607773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.608635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=146238397f514eada174ab3374be5418) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:37.608932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.610629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.626442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 318/665] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/role_assignments?user.id=146238397f514eada174ab3374be5418&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:37.626442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.654182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d9e10-9516-43a4-a3e8-7609069bda82 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.658852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.716689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.717373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.717755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.718097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.751713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754546bf-11a4-45f0-b8fa-7946a251267d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:37.778899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.796383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:37.796889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.797179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:37.797529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:37.798450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:37.799159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.878107 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:37.879633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:37.911307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:37.921973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 348/666] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:37.923702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:37.994713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 319/667] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:37.994713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.014514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:38.030688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246426ce-fcc3-45dd-9904-48bc22aa4ff2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.068874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.069494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.069773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.070046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.070797 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=40e12aa547b84ac38fd222a9f5f62ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:38.071102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.072533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.090444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 349/668] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/role_assignments?user.id=40e12aa547b84ac38fd222a9f5f62ae2&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:38.090444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.113715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ce0c80-1af4-4105-99b8-34fc866393d0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.150056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:38.161896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.163112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.163649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.165282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.180364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:38.187895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:38.188963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:38.189794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:38.190442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 320/669] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:38.191031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.226213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f3f2ff-6a4f-413a-9683-0f371cd9bcca None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.267153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.268010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.268355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.268628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.281496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.283935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.294392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:38.300789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:38.309440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:38.311883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:38.311883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:38.311883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:38.311883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 350/670] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:38.311883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.313841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef41d43-a668-49ff-8e02-5c859dae9adc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.362256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.362920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.363244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.363453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.375393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.377037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:38.393870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:38.395015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:38.395898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:38.395898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:38.395898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:38.395898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 321/671] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:38.395898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.435593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af975555-c7b9-4cb4-8631-7f757a146432 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.448969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:38.478598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.479221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.479511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.479785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.480459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:38.480948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.482343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.493978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 351/672] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:38.495153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.510273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dc9e3e-732f-4d0f-8e5f-2a0e3a3b3d1f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.555265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.555788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.556064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.556333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.556971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:38.557267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.558663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.572738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 322/673] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:38.578108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.585546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:38.646862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0a6925-83f3-4767-9fe8-a032f0ea60c1 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.686768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:38.690972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af15951-4859-4c4d-9fd8-6d72a13ac2ae None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.700338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.701654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.702635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.703397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.704721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:38.705261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.707547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.730295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 352/674] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:38.730499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.730499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.730680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.730761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.730761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.742312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.744094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:38.760040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:38.761001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:38.761876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:38.761876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:38.761876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:38.761876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 323/675] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:38.762643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.867054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cefdcce-74d0-45a1-aab1-e70f69f2c0c5 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.871801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5fb34f-daa1-41bf-a143-921b8e9eedc5 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:38.906809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:38.909293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.909923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.910196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.910496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.917852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.919322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.919951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:38.924768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:38.924768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:38.924768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:38.930267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:38.931816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:38.932866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:38.934063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:38.934575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:38.937711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 324/676] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:38.938767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:38.947875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:38.948905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:38.949920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:38.954736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 353/677] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:38.955700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:38.962365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:39.069003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7f7433-cd2e-442b-b8cd-e799258d8758 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.086868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f996a5-a3ff-45d7-b974-2dab0a155f8d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.111468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.112172 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.112536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.112876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.113634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:39.114003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.115522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:39.132027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.132512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 354/678] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:39.132621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.132956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.133363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.134028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:39.134144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:39.134444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.135880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:39.153584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 325/679] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:39.154502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:39.193202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:39.235780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:39.262808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1ba66b-9648-440a-af7e-43d780668530 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.267444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dfe9c0-2cd6-4ea7-aa66-3bfcabf67492 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.300933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.301976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.302533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.303047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.303454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.304126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.304327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:39.304530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.304879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.305202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.305768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=f49a19f9484d4aa99567e66b4fc396ef) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:39.306138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.307269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:39.307823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:39.314826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 326/680] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:39 2022] GET /identity/v3/role_assignments?user.id=f49a19f9484d4aa99567e66b4fc396ef&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:39.315757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:39.320983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 355/681] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:39.327315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:39.436674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afee548a-c814-490a-a138-2934f3b7bd2b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.453252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350163a6-0178-4ba0-8713-2b3ec369cc8f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.455733 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:39.488343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.489918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.490561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.491080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] PATH_INFO: `/v3/users/swift` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.494177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.495016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.496058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:39.502284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.502284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.502284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=df391d2bf7884538823f04db5564d02c) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:39.502284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.502284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:39.509133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 356/682] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:39 2022] GET /identity/v3/role_assignments?user.id=df391d2bf7884538823f04db5564d02c&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:39.510363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.511997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:39.534334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:39.535561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:39.536708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:39.543579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 327/683] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:39.550256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:39.687925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c3b5a8-b4a8-49d3-9091-b06c60b181ac None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.727687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.728549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.729004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.729427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.730301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:39.730772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.732332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:39.752421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 357/684] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Oct 4 08:33:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:39.757847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:39.864316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916fb2f-63a8-4e03-8ed2-ef8b0dbd7b4c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:39.901111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7468b0fa-3b58-4d69-b7fe-91bd09376ffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:39.901995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468b0fa-3b58-4d69-b7fe-91bd09376ffe None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:39.902460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468b0fa-3b58-4d69-b7fe-91bd09376ffe None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:39.902881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468b0fa-3b58-4d69-b7fe-91bd09376ffe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:39.979398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:39.981557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7468b0fa-3b58-4d69-b7fe-91bd09376ffe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:39.983373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7468b0fa-3b58-4d69-b7fe-91bd09376ffe None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:40.111027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 328/685] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:40.116461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:40.148294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:40.179155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2f9d1f-4338-4ce3-b7d8-29dc840ce7c1 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:40.179433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2f9d1f-4338-4ce3-b7d8-29dc840ce7c1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:40.179707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2f9d1f-4338-4ce3-b7d8-29dc840ce7c1 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:40.182423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 329/686] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:40.186569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcd4a6c-2494-4eb8-934f-06e17145ae66 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:40.186848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcd4a6c-2494-4eb8-934f-06e17145ae66 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:40.187112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcd4a6c-2494-4eb8-934f-06e17145ae66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:40.214217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7fcd4a6c-2494-4eb8-934f-06e17145ae66 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:40.215442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7fcd4a6c-2494-4eb8-934f-06e17145ae66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:40.274325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7863ce-7800-41f1-8bbc-613b59072d13 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:40.329292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:40.330116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:40.330390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:40.330689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:40.342316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:40.343803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:40.362594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:40.363581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:40.364442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:40.364442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:40.364442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:40.364621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 358/687] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:40.377336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:40.377516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 330/688] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:40.490821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:40.528809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcd74d3-0ccc-45ac-a1ea-4e8c8dbaaa29 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:40.569220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61962a2f-99d5-446b-8204-86429af83049 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:40.569628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61962a2f-99d5-446b-8204-86429af83049 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:40.569812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61962a2f-99d5-446b-8204-86429af83049 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:40.571786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 331/689] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:40.579055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a4ad4-2823-40b8-8833-b10cfcf9e09c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:40.579055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a4ad4-2823-40b8-8833-b10cfcf9e09c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:40.579055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a4ad4-2823-40b8-8833-b10cfcf9e09c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:40.592146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:40.593264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:40.593768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:40.594251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:40.595517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:40.596024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:40.597712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:40.611897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 359/690] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:40.621218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-186a4ad4-2823-40b8-8833-b10cfcf9e09c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:40.621218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-186a4ad4-2823-40b8-8833-b10cfcf9e09c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:40.622222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:40.667306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:40.753640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 332/691] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:40.766037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186a4ad4-2823-40b8-8833-b10cfcf9e09c None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:40.820911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af465815-b803-4383-afe9-d1dcdaca59da None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:40.895163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:40.895814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:40.896085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:40.897682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:40.905695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:40.923273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:40.924773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:40.946191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:40.947165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:40.948025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:40.949867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186a4ad4-2823-40b8-8833-b10cfcf9e09c None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:40.952315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 360/692] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:40.959999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.020475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.021481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.021958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.022468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.033982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.035710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:41.070142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:41.071288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:41.072142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:41.072142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:41.072142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:41.072839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 333/693] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:41.078261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.096721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:41.118585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0da05b-c288-4a58-bd9e-ba0ea49ee215 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.166586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.168286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.168768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.169187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.170022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:41.170491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.172077 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.191528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 361/694] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:41.192627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.207346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6febbd-112c-4093-ae28-f297130c855a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.246674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.247567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.247988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.248452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.261019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:41.268131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:41.269726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:41.270963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:41.271958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:41.274210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 334/695] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:41.275176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.334646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dec32e-53f1-47fa-93d0-855ab796e43e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.355211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:41.379676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.380692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.381093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.381478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.382351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:41.382764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.384520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.407090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 362/696] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:41.408093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.409343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003140c-5a70-40f9-8e9f-db8d36d90c70 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.447892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.448847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.449298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.449739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.450752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=085a0cc3f5cb477989e6e75adea7c53a) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:41.451233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.452829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.460531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 335/697] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:41 2022] GET /identity/v3/role_assignments?user.id=085a0cc3f5cb477989e6e75adea7c53a&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:41.461693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.525202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498da9eb-7198-4eef-9084-03c849151c3f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.544600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:41.565228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.566021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.566356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.578079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.578079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:41.578079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.578079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.578398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 363/698] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:41.585495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.598054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f575962-bc02-4f6e-bc57-d8a39ee1144a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.635617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:41.643475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.644483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.644971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.645425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.646264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.648791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.716158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b712fea0-79aa-46a1-b699-0be4aa7f0ea2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.757267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.758226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.758703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.759148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.771681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 336/699] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:33:41 2022] POST /identity/v3/users => generated 254 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:41.784419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.786056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:41.806838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:41.807888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:41.808762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:41.808762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:41.808762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:41.808762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 364/700] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:41.815558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:41.935732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323e6719-2c3f-4314-aad5-d7c0fe0774dc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:41.974261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f8f3e-0112-441d-b389-bff5257af340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:41.975284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f8f3e-0112-441d-b389-bff5257af340 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:41.975796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f8f3e-0112-441d-b389-bff5257af340 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:41.976268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f8f3e-0112-441d-b389-bff5257af340 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:41.977133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9f8f3e-0112-441d-b389-bff5257af340 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:41.977615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9f8f3e-0112-441d-b389-bff5257af340 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:41.979270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9f8f3e-0112-441d-b389-bff5257af340 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:41.995824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 337/701] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:42.002249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:42.044293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:42.120267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae6b459-928f-4221-a941-5fbfeeaa498d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:42.126749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300c4e3-ac69-466e-a128-e2ff8a6e3765 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.127326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300c4e3-ac69-466e-a128-e2ff8a6e3765 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.127825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300c4e3-ac69-466e-a128-e2ff8a6e3765 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.129669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 338/702] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:42.134773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b1c93c-6638-4281-bf39-b6a291a84769 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.135248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b1c93c-6638-4281-bf39-b6a291a84769 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.135519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b1c93c-6638-4281-bf39-b6a291a84769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.160634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:42.160962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-02b1c93c-6638-4281-bf39-b6a291a84769 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:42.161800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.162166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-02b1c93c-6638-4281-bf39-b6a291a84769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:42.162571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.163122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.171634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:42.173430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:42.185660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:42.186842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:42.187761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:42.191086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 365/703] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:33:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:42.198607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:42.278759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:42.300613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:42.310581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4acdc6-700c-4365-b576-f37f08b355c0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:42.336599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 339/704] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:42.343538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0fba3b-c79a-4dbf-ab2b-1eb240f2c21c None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.343826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0fba3b-c79a-4dbf-ab2b-1eb240f2c21c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.344101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0fba3b-c79a-4dbf-ab2b-1eb240f2c21c None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.345811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 340/705] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:42.354559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ebf2fa-90e2-404d-803c-a4a61b6ab4c7 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.354838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ebf2fa-90e2-404d-803c-a4a61b6ab4c7 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.355110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ebf2fa-90e2-404d-803c-a4a61b6ab4c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.389723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:42.390341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.390652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.390924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.392201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:42.392501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:42.393936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:42.396630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-70ebf2fa-90e2-404d-803c-a4a61b6ab4c7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:42.397841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-70ebf2fa-90e2-404d-803c-a4a61b6ab4c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:42.424649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 366/706] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:33:42 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:42.432509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:42.529872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 341/707] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:42.597152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f46b23-699d-4e4d-b49d-471a25da1489 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:42.632238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:42.632849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.633125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.633393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.656248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:42.661212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ebf2fa-90e2-404d-803c-a4a61b6ab4c7 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:42.716469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:42.717928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:42.835243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 367/708] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:42.844698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:42.886652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ebf2fa-90e2-404d-803c-a4a61b6ab4c7 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:42.896837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:42.906165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:42.926420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:42.927074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:42.927349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:42.927637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:42.928192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:42.929823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:42.941942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:42.942995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:42.943813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 342/709] 10.222.0.27 () {56 vars in 1129 bytes} [Tue Oct 4 08:33:42 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:42.945214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:42.965639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654f4c5b-7002-4d6c-a8fc-eef2f26ef412 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.010473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.011290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.011639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.011918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.023761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.025290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:43.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:43.043033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:43.045180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:43.045180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:43.045180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:43.050435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 368/710] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:43.050435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.054336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ded6b08-6d4f-4c71-9c8f-87a8925e9e02 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.091593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.092137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.092409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.092677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] PATH_INFO: `/v3/services/network` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.093248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.094768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:43.105094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:43.107403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:43.109432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:43.114513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 343/711] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:42 2022] GET /identity/v3/services/network => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:43.114513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.153968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:43.164361 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:43.168399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96960e3f-cadb-4c55-89e2-a6043af5539a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.207340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21be7109-675b-4797-9c92-23e720041217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.207961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21be7109-675b-4797-9c92-23e720041217 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.208240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21be7109-675b-4797-9c92-23e720041217 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.208508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21be7109-675b-4797-9c92-23e720041217 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.209142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21be7109-675b-4797-9c92-23e720041217 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:43.209560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21be7109-675b-4797-9c92-23e720041217 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.211254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21be7109-675b-4797-9c92-23e720041217 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.216510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 369/712] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:43.217406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21be7109-675b-4797-9c92-23e720041217 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.227751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5969d870-8a00-4168-aba2-042b01d295a3 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.265218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.266419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.266899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.267362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.268209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:43.268733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.270419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.282900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 344/713] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:43.284191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.326621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21be7109-675b-4797-9c92-23e720041217 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.366938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.368574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.369100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.369516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.370462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] RBAC: Adding query filter params (name=network) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:43.370929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.372991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.383366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 370/714] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/services?name=network => generated 124 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:43.383547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.391976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:43.399148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d77c79-fa09-49e8-9b17-6bf6f8d387ef None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.414877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:43.437351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.438382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.438901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.439390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.440215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:43.440712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.442279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.449918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 345/715] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:43.451143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.501464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7429fce3-4a1b-4131-89ae-47bc0b3f0370 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.574102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b87c9-6caf-4247-a358-d761327ac3c1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.579253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.579922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.580206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.580479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.596832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.598298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.614967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.616086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.616569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.617171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.618009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] RBAC: Adding query filter params (type=network) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:43.618797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.620372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:43.624131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:43.624992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:43.627114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 371/716] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:43.634559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 346/717] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/services?type=network => generated 124 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:43.640265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.787189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:43.801685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa20d98a-92f0-4026-84bb-0c9c77ae7f5e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:43.839213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c94f73-a869-4ace-89ca-49e16627d049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:43.839824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c94f73-a869-4ace-89ca-49e16627d049 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:43.840102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c94f73-a869-4ace-89ca-49e16627d049 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:43.840377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c94f73-a869-4ace-89ca-49e16627d049 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:43.841065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c94f73-a869-4ace-89ca-49e16627d049 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:43.841371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c94f73-a869-4ace-89ca-49e16627d049 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:43.842869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c94f73-a869-4ace-89ca-49e16627d049 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:43.860848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 372/718] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:43.866437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:43.910190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:43.991246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaad4d6-0d4f-4902-8c4c-052047636c04 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:44.029922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fb252a-f021-40e8-9fa0-1fc2500815b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:44.030946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fb252a-f021-40e8-9fa0-1fc2500815b5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:44.031431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fb252a-f021-40e8-9fa0-1fc2500815b5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:44.031964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fb252a-f021-40e8-9fa0-1fc2500815b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:44.034149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fb252a-f021-40e8-9fa0-1fc2500815b5 None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=40e12aa547b84ac38fd222a9f5f62ae2) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:44.034711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fb252a-f021-40e8-9fa0-1fc2500815b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:44.036359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fb252a-f021-40e8-9fa0-1fc2500815b5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:44.046999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 347/719] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:43 2022] GET /identity/v3/role_assignments?user.id=40e12aa547b84ac38fd222a9f5f62ae2&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:44.539110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788b81a9-1c63-4dfa-af43-7c23c0bc4e2a None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:44.539801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788b81a9-1c63-4dfa-af43-7c23c0bc4e2a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:44.540289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788b81a9-1c63-4dfa-af43-7c23c0bc4e2a None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:44.542105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 373/720] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:44.547265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa34616-927e-4612-9638-a94288e4dd71 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:44.548488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa34616-927e-4612-9638-a94288e4dd71 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:44.548986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa34616-927e-4612-9638-a94288e4dd71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:44.565031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76101957-db8d-4a34-b7d8-63f1454767e7 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:44.565345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76101957-db8d-4a34-b7d8-63f1454767e7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:44.565590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76101957-db8d-4a34-b7d8-63f1454767e7 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:44.567321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 374/721] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:44.573818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a8a6b4-7c7f-4520-89ab-7bcb230b5b21 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:44.573818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a8a6b4-7c7f-4520-89ab-7bcb230b5b21 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:44.573818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a8a6b4-7c7f-4520-89ab-7bcb230b5b21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:44.575388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-efa34616-927e-4612-9638-a94288e4dd71 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:44.577366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-efa34616-927e-4612-9638-a94288e4dd71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:44.593457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c9a8a6b4-7c7f-4520-89ab-7bcb230b5b21 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:44.594761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c9a8a6b4-7c7f-4520-89ab-7bcb230b5b21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:44.706861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 375/722] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:33:44.716032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 348/723] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:45.189854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a8a6b4-7c7f-4520-89ab-7bcb230b5b21 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:45.262428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa34616-927e-4612-9638-a94288e4dd71 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:45.285069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:45.307751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:45.309935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a8a6b4-7c7f-4520-89ab-7bcb230b5b21 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:45.348140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:45.348820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:45.349516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:45.349797 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:45.369211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:45.375006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:45.375963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:45.376771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:45.386430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 376/724] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:45.386430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:45.432098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa34616-927e-4612-9638-a94288e4dd71 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:45.507827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:45.508461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:45.508735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:45.508999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:45.531658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:45.537120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:45.538110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:45.539020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:45.539648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c0435-8286-44d3-a148-db80890d66ca None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:45.551647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 349/725] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:45.553086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:45.603319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:45.603933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:45.604209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:45.604478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:45.605139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:45.605438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:45.607512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:45.620910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:45.631844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 377/726] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:45.633151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:45.729261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:45.772468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2e0c45-9b10-496e-ae7c-96fccc7b2aae None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:45.833415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0723f76a-b662-4d2b-92ff-9baef92e9d8a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:45.839348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:45.839958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:45.840236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:45.840506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:45.841171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:45.841475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:45.842865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:45.860452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 350/727] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:45.866756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:45.896217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:45.896862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:45.897147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:45.897419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:45.909016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:45.910582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:45.927037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:45.928001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:45.928851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:45.928851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:45.928851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:45.928851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 378/728] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:45.940248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:45.994087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375f59e6-5a6a-42b2-9886-a03cd70ff6d0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.036519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.037069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.037345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.037611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.038381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.055608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.057277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:46.058828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:46.084428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:46.086001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:46.086906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:46.086906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:46.086906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:46.086906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 351/729] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:46.094023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:46.140033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c3b984-7b54-4a22-9f14-0724a055c7d7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.201010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3776783-0e33-4994-9bf3-5a55a79784dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.201619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3776783-0e33-4994-9bf3-5a55a79784dd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.201892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3776783-0e33-4994-9bf3-5a55a79784dd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.202161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3776783-0e33-4994-9bf3-5a55a79784dd None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.202848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3776783-0e33-4994-9bf3-5a55a79784dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:46.203150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3776783-0e33-4994-9bf3-5a55a79784dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.205098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3776783-0e33-4994-9bf3-5a55a79784dd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:46.222143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 379/730] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:46.223374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a6a44-c583-4987-909a-7a63643e0cbe None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.223816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a6a44-c583-4987-909a-7a63643e0cbe None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.224085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556a6a44-c583-4987-909a-7a63643e0cbe None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.225485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 380/731] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:46.227493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bee5ed-de54-4b9d-9715-70bcb2bed11b None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.228003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bee5ed-de54-4b9d-9715-70bcb2bed11b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.228469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bee5ed-de54-4b9d-9715-70bcb2bed11b None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.229865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 381/732] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:46.230889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bee5ed-de54-4b9d-9715-70bcb2bed11b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:46.262205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b26926-2bd0-4604-93ec-60161e790f94 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.299032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5411717-128a-4f67-9382-26351da25bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.299556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5411717-128a-4f67-9382-26351da25bb4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.299824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5411717-128a-4f67-9382-26351da25bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.300098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5411717-128a-4f67-9382-26351da25bb4 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.300734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5411717-128a-4f67-9382-26351da25bb4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:46.301031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5411717-128a-4f67-9382-26351da25bb4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.302390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5411717-128a-4f67-9382-26351da25bb4 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:46.314927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 352/733] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:46.316188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b524e9-1646-4fb1-800a-b3a83afa6043 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.316470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b524e9-1646-4fb1-800a-b3a83afa6043 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.316736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b524e9-1646-4fb1-800a-b3a83afa6043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.336689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f4b524e9-1646-4fb1-800a-b3a83afa6043 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:46.338209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f4b524e9-1646-4fb1-800a-b3a83afa6043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:46.348576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bee5ed-de54-4b9d-9715-70bcb2bed11b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.362885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.390610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbe4a89-b0b7-4786-a963-8d3429849332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.390610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe4a89-b0b7-4786-a963-8d3429849332 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.390610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe4a89-b0b7-4786-a963-8d3429849332 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.390610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe4a89-b0b7-4786-a963-8d3429849332 None admin] PATH_INFO: `/v3/users/swift` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.395373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.397930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbe4a89-b0b7-4786-a963-8d3429849332 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.399471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbe4a89-b0b7-4786-a963-8d3429849332 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-5dbe4a89-b0b7-4786-a963-8d3429849332 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:46.412176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:46.413092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 382/734] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:46 2022] GET /identity/v3/users/swift => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79748b7-98be-4a2d-85ac-efa5d5b5bdeb None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.414767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79748b7-98be-4a2d-85ac-efa5d5b5bdeb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.415728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79748b7-98be-4a2d-85ac-efa5d5b5bdeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.426455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 353/735] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:46.426455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b524e9-1646-4fb1-800a-b3a83afa6043 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:46.439908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a79748b7-98be-4a2d-85ac-efa5d5b5bdeb None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:46.441307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a79748b7-98be-4a2d-85ac-efa5d5b5bdeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:46.529719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 383/736] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:46.530913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79748b7-98be-4a2d-85ac-efa5d5b5bdeb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:46.564830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b524e9-1646-4fb1-800a-b3a83afa6043 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.641512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eab483-dfb2-44eb-8891-9df1d39867b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.642452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eab483-dfb2-44eb-8891-9df1d39867b7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.642900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eab483-dfb2-44eb-8891-9df1d39867b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.643306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eab483-dfb2-44eb-8891-9df1d39867b7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.652399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eab483-dfb2-44eb-8891-9df1d39867b7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.653848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eab483-dfb2-44eb-8891-9df1d39867b7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-90eab483-dfb2-44eb-8891-9df1d39867b7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:46.668168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:46.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:46.670233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 354/737] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:33:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:46.670864 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c7ad9f-3956-474b-947b-d141edc46e09 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.671101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c7ad9f-3956-474b-947b-d141edc46e09 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.671313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c7ad9f-3956-474b-947b-d141edc46e09 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.674527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 355/738] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:46.679172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c7ad9f-3956-474b-947b-d141edc46e09 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:46.692656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79748b7-98be-4a2d-85ac-efa5d5b5bdeb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.694965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.727105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70984bc1-f70c-49d3-92e2-861e1fb66dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.727759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70984bc1-f70c-49d3-92e2-861e1fb66dc2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.728042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70984bc1-f70c-49d3-92e2-861e1fb66dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.728307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70984bc1-f70c-49d3-92e2-861e1fb66dc2 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.729030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70984bc1-f70c-49d3-92e2-861e1fb66dc2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:46.729329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70984bc1-f70c-49d3-92e2-861e1fb66dc2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.729692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.730837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70984bc1-f70c-49d3-92e2-861e1fb66dc2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:46.750204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 384/739] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Oct 4 08:33:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:46.751908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55662176-aec0-45b4-ae47-f036a245f587 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.752250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55662176-aec0-45b4-ae47-f036a245f587 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.752593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55662176-aec0-45b4-ae47-f036a245f587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.771099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-55662176-aec0-45b4-ae47-f036a245f587 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:46.772277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-55662176-aec0-45b4-ae47-f036a245f587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:46.795607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c7ad9f-3956-474b-947b-d141edc46e09 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.830668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.831147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.831425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.831698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.832433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:46.832738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.834530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:46.842838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 385/740] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:33:46.843782 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55662176-aec0-45b4-ae47-f036a245f587 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:46.858452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 356/741] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Oct 4 08:33:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:46.858452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:46.935025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.945246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55662176-aec0-45b4-ae47-f036a245f587 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.967156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.971149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4997b18d-cf53-417c-9b21-ad5aec6e504a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:46.985676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:46.986220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:46.986594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:46.986873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:46.987524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:46.988885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:47.010072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:47.010699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:47.010973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:47.011236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:47.077569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:47.080071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:47.095191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 386/742] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:33:46 2022] POST /identity/v3/services => generated 240 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:47.096569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.210170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:47.220130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 357/743] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:47.220130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.245936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713f70b-8717-497d-96a4-03f76a26bd6b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:47.285136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:47.285752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:47.286027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:47.286350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:47.286978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:47.288348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:47.331408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d19ef5-52e6-480d-a6c8-333be1e7172f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:47.365161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:47.365697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:47.365974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:47.366244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:47.426964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 387/744] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:33:47 2022] POST /identity/v3/services => generated 251 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:47.427948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.443479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:47.444881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:47.448222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:47.448579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:47.583759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 358/745] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:47 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:33:47.584735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.597696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cb0000-b6a6-4b05-b3c4-4973594906e7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:47.641965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:47.643145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:47.643473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:47.643646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:47.656354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:47.661745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:47.662698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:47.663521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 388/746] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:47.664188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.703543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:47.706809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99c9a60-24fc-4f9c-8624-63a8b5a6c010 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:47.741740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:47.742381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:47.742725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:47.743096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:47.749683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:47.756063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:47.757915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:47.773797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:47.774918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:47.775835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:47.775835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:47.775835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:47.775835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 359/747] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:47.776556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.801187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1826f2e2-4795-4041-9ba3-3dde163a058e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:47.836927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:47.837601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:47.837882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:47.838151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:47.852211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:47.853698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:47.870607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:47.873182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:47.875721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:47.875721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:47.875721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:47.875721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 389/748] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:47.875721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.903175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e241232-9631-46f9-8710-0b683076de8d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:47.943173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:47.943957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:47.944238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:47.945011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:47.945690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:47.946276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:47.947758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:47.961516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 360/749] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:47.962617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:47.981411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:47.983832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f01258b-8bdf-46aa-8743-7a4d47eb2451 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.018854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:48.027892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.029499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.030040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.030480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.031698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:48.032803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.035076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.052740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 390/750] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:48.053873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.086191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f6100-49b0-48e2-9c9e-f7971bec8eea None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.126988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.127745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.128211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.128499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.129238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:48.129709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.131136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.147255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 361/751] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:48.148192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.189880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c30d22a-2365-479f-a7a7-0bfd58fbd017 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.229443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:48.238879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.239901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.240611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.246908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.255813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.256529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b398b3a-f2f8-4ff6-b674-ca45a233267a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.257616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:48.276422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:48.277629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:48.278743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:48.278743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:48.278743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:48.285248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 391/752] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:48.285817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.310667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.311857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.312707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.313681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.315787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:48.325237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.326986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:48.342831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:48.343922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:48.345008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:48.347552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 362/753] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:48.348557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.407078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d445edfb-fe56-4e3c-95d5-38326e4276af None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.449484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.451397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.451971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.451971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.461528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.463133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.474729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fcc7e4-ffe5-44db-96a0-91ac840f90ad None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:48.475713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:48.476541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:48.477430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:48.481632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 392/754] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:48.482957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.540332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.541341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.541828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.542298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.543163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:48.543817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.545469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.549129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:48.571400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 363/755] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:48.572582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.577238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:48.592784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a82f8c7-cf00-4e20-acc3-c52cd4770c00 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.629982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.630618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.630898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.631167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.631872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:48.632153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.633681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.650760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 393/756] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:48.651633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.753850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b99c1-c59c-4d9f-9e9d-3f68bc1386ec None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.772443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f232f9af-07a9-414d-9ba8-def0e5fcaa15 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:48.815018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.815586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.815869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.816145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.823785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.825743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:48.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:48.841857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:48.842783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:48.843837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:48.843837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:48.843837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:48.844066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:48.845302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:48.845786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:48.847390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 394/757] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:48.848260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.849563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:48.865742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:48.882589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 364/758] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:48.883506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:48.986393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf81cff6-288b-496f-ab92-2c64df763475 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:49.014278 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:49.061596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:49.062622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.063155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.063692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.064877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=085a0cc3f5cb477989e6e75adea7c53a) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:49.066957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:49.070354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:49.090058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 395/759] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/role_assignments?user.id=085a0cc3f5cb477989e6e75adea7c53a&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:49.090058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:49.258833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:49.324567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcbe0f8-ed76-4074-acfc-8f8a94cf9447 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:49.343630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2fddac-e2c6-453d-b024-4ac95d8ce70f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:49.369036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd658a0-80b0-4e57-bc7b-c5049c48646e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:49.369662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd658a0-80b0-4e57-bc7b-c5049c48646e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.369941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd658a0-80b0-4e57-bc7b-c5049c48646e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.370214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd658a0-80b0-4e57-bc7b-c5049c48646e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.370979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd658a0-80b0-4e57-bc7b-c5049c48646e None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=c9cff7d749314dbc94645a7884792496) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:49.371285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd658a0-80b0-4e57-bc7b-c5049c48646e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:49.374647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd658a0-80b0-4e57-bc7b-c5049c48646e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:49.386444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 396/760] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:49 2022] GET /identity/v3/role_assignments?user.id=c9cff7d749314dbc94645a7884792496&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 259 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:49.386444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f72ec3-36bd-48df-a901-4bd4f45e6433 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.386444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f72ec3-36bd-48df-a901-4bd4f45e6433 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.386444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f72ec3-36bd-48df-a901-4bd4f45e6433 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.386875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 397/761] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:49.392751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96209a6e-0fee-4b8f-beb9-689e32239d2c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.393115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96209a6e-0fee-4b8f-beb9-689e32239d2c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.394961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96209a6e-0fee-4b8f-beb9-689e32239d2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.401073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:49.403057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.403524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.403973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.404886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:49.405392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:49.411318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:49.425310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-96209a6e-0fee-4b8f-beb9-689e32239d2c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:49.426584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-96209a6e-0fee-4b8f-beb9-689e32239d2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:49.453033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 365/762] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:33:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 570 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:49.459114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:49.540083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 398/763] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:33:49.658801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca81506-58b4-4a2c-ab6c-31dbf3b79e6d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:49.673646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:49.706351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beff7b8-c28a-4a07-a1dd-c1042d241319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:49.707008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beff7b8-c28a-4a07-a1dd-c1042d241319 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.707286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beff7b8-c28a-4a07-a1dd-c1042d241319 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.707557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beff7b8-c28a-4a07-a1dd-c1042d241319 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.724451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4beff7b8-c28a-4a07-a1dd-c1042d241319 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:49.725904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4beff7b8-c28a-4a07-a1dd-c1042d241319 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-4beff7b8-c28a-4a07-a1dd-c1042d241319 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:49.746587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:49.747580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:49.748460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:49.748460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:49.748460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:49.748460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 366/764] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:49.753880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96209a6e-0fee-4b8f-beb9-689e32239d2c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:49.759173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269700bc-8671-422c-8c57-f3ac46d5ee31 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.759645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269700bc-8671-422c-8c57-f3ac46d5ee31 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.759724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269700bc-8671-422c-8c57-f3ac46d5ee31 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.761257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 367/765] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:49.766077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a583cbc0-393c-424c-ad75-b214470ea67f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.766995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a583cbc0-393c-424c-ad75-b214470ea67f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.767267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a583cbc0-393c-424c-ad75-b214470ea67f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.787108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a583cbc0-393c-424c-ad75-b214470ea67f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:49.788283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a583cbc0-393c-424c-ad75-b214470ea67f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:49.842456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:49.894134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 368/766] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:33:49.894694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96209a6e-0fee-4b8f-beb9-689e32239d2c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:49.895926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a583cbc0-393c-424c-ad75-b214470ea67f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:49.972565 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:49.973449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:49.973449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:49.973623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:49.974262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:49.974590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:49.975964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.004567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 399/767] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:50.006587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:50.010233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.013916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a583cbc0-393c-424c-ad75-b214470ea67f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.060848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.061855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.062341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.062857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] PATH_INFO: `/v3/services/network` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.064218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.065794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:50.087754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:50.088865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:50.089688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:50.092429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 369/768] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:49 2022] GET /identity/v3/services/network => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:50.103423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.144610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c64306-2711-4777-89f0-8f1779ee645f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.169718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:50.192322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.193098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.193384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.193657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.213986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.215634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:50.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:50.236165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:50.237059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:50.239655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2319b1a8-12ca-4c15-b69f-8d3122ab2419 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.242668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 400/769] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:33:50.243955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.276935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.277541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.277816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.278187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.278848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] RBAC: Adding query filter params (name=network) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:50.279145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.280505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.285894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 370/770] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/services?name=network => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:50.286606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.320372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:50.388473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccef23a-d8db-4d60-a69a-726ef1af1d94 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.413344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ad1f5-e15a-4856-bd38-e502630cadde None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.429674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.430673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.430969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.431248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.432214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:50.432525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.434186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.450913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 401/771] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:50.451772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.455544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.456622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.457062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.457621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.458228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.459729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:50.470596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:50.471655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:50.472501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 371/772] 10.222.0.27 () {56 vars in 1129 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:50.473289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.519313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:50.577871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:50.595967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797165c2-f5bd-48b5-a57d-462f0bfc9fa0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.596343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81036ab9-7409-438f-a3ec-e0c8e5c7d131 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.639320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.639935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.640213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.640479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.641157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] RBAC: Adding query filter params (type=network) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:50.642006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.642472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.643411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.643517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.644028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.644790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/40e12aa547b84ac38fd222a9f5f62ae2/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.647302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=40e12aa547b84ac38fd222a9f5f62ae2, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.648787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.672661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 402/773] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/services?type=network => generated 350 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:50.672661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.736402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 372/774] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:50 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/40e12aa547b84ac38fd222a9f5f62ae2/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:50.736402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.817319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042b9366-b752-4116-a185-864ff1377b6a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.828047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:50.880615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.881245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.881522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.881789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:50.882439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:50.882750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:50.884551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:50.900845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 403/775] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:50.906154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:50.946462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e700bfd1-aa91-47f7-8404-d9321347ad46 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:50.997094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:50.998644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:50.999172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:50.999587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.000581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] RBAC: Adding query filter params (interface=public, service_id=cc75a85996a74a0fbcb49f5eea16f07a, region_id=RegionOne) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:51.001039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:51.002645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:51.018041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 373/776] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/endpoints?service_id=cc75a85996a74a0fbcb49f5eea16f07a&interface=public®ion_id=RegionOne => generated 194 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:51.019752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:51.038545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:51.070972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ee6474-311b-4605-a63e-2054d5572f37 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:51.109578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:51.110502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.110929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.111347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.112117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:51.112659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:51.114395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:51.120376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 404/777] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Oct 4 08:33:50 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:51.126056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:51.137805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d280523-e452-4633-a3dc-81130c340158 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:51.188846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:51.189852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.190463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.190945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.191404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:51.193003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:51.194740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:51.200924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 374/778] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:33:51 2022] GET /identity/v3/services => generated 772 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:51.294421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce4d170-b9e3-4d51-b420-6b4736926be6 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:51.331808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095d0f64-295d-4b36-a033-680d0195d24b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:51.332345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d0f64-295d-4b36-a033-680d0195d24b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.332620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d0f64-295d-4b36-a033-680d0195d24b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.332887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d0f64-295d-4b36-a033-680d0195d24b None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.333593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095d0f64-295d-4b36-a033-680d0195d24b None admin] RBAC: Adding query filter params (interface=public, service_id=990564e2ba7741a0a2649e8c387ff080, region_id=RegionOne) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:51.333896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095d0f64-295d-4b36-a033-680d0195d24b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:51.335292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095d0f64-295d-4b36-a033-680d0195d24b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:51.343867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 405/779] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:33:51 2022] GET /identity/v3/endpoints?service_id=990564e2ba7741a0a2649e8c387ff080&interface=public®ion_id=RegionOne => generated 194 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:51.349832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:51.472334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:51.498731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bda03b9-3845-41ca-ae7b-4b48ab0e21c1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:51.547220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834ae2e9-fa6e-4b41-b8df-33df64f9ba7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:51.548424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834ae2e9-fa6e-4b41-b8df-33df64f9ba7b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.548701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834ae2e9-fa6e-4b41-b8df-33df64f9ba7b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.548974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834ae2e9-fa6e-4b41-b8df-33df64f9ba7b None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.549577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834ae2e9-fa6e-4b41-b8df-33df64f9ba7b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:51.550966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834ae2e9-fa6e-4b41-b8df-33df64f9ba7b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:51.556436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 375/780] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:33:51 2022] GET /identity/v3/services => generated 772 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:51.737308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fca462-1dca-4dd9-b319-175cbe898c05 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.737832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fca462-1dca-4dd9-b319-175cbe898c05 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.738252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fca462-1dca-4dd9-b319-175cbe898c05 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.740162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 406/781] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:51.746393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6645395-ac0b-4285-bba8-21b493e0591d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.746730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6645395-ac0b-4285-bba8-21b493e0591d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.746966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6645395-ac0b-4285-bba8-21b493e0591d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.775669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e6645395-ac0b-4285-bba8-21b493e0591d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:51.776996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e6645395-ac0b-4285-bba8-21b493e0591d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:51.868582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 376/782] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:33:51.895955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e1cab5-3954-43ea-b35d-2dafc339ef15 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.896592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e1cab5-3954-43ea-b35d-2dafc339ef15 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.897122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e1cab5-3954-43ea-b35d-2dafc339ef15 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.898992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 407/783] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:51.904015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de173f-10fa-4ac5-93ba-1c70098ec4bc None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:51.904296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de173f-10fa-4ac5-93ba-1c70098ec4bc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:51.904567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de173f-10fa-4ac5-93ba-1c70098ec4bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:51.927790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a2de173f-10fa-4ac5-93ba-1c70098ec4bc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:51.928992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a2de173f-10fa-4ac5-93ba-1c70098ec4bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:52.027084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:52.033164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 377/784] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:33:52.158524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e1cab5-3954-43ea-b35d-2dafc339ef15 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:52.294588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:52.298750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e1cab5-3954-43ea-b35d-2dafc339ef15 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:52.351760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:52.352389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:52.352663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:52.352927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:52.371142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:52.372781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2de173f-10fa-4ac5-93ba-1c70098ec4bc None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:52.376552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:52.377589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:52.379384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 408/785] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:52.386659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:52.568746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce61f6b-ba41-460e-a5ff-d3867ce780d8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:52.589269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2de173f-10fa-4ac5-93ba-1c70098ec4bc None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:52.629191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:52.630103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:52.630576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:52.631037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:52.632368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:52.632954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:52.634505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:52.644541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 409/786] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:52.650157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:52.681856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:52.684284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:52.685906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:52.687082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:52.687891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:52.711491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:52.716551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:52.717468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:52.718269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:52.720884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 378/787] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:52.729431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:52.792756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:52.837664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44217409-5ff5-4460-94af-434fb86b1d42 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:52.877958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b143c33-971d-493d-8fab-b845eb59ac46 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:52.878995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:52.879846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:52.881174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:52.881639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:52.882511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:52.883644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:52.885914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:52.899245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 379/788] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:52.906218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:52.950196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:52.951251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:52.952760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:52.953248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:52.980525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:52.982011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:53.007862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:53.008834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:53.009692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:53.009692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:53.009692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:53.018513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 410/789] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 369 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:53.022695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:53.071787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3b2642-c9ba-4fd9-8e94-983e658c5156 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:53.093390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:53.111516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:53.112515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.113049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.113522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.124892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:53.127434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:53.143973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:53.144926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:53.145782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:53.145782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:53.145782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:53.148802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 380/790] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:53.154076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:53.176473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:53.227082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97332843-659d-48a5-bb31-49a6b243c9f3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:53.302803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30eb0e-a602-4960-bfa5-0a170ae1b19c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:53.311114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:53.311743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.312031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.312310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.313522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:53.314312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:53.315817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.340426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 411/791] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:53.346975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:53.351499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:53.352281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.352723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.353144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.353922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:53.354362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:53.356775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.399300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 381/792] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:53.407275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:53.468405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:53.548772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5120cb-baac-4e9a-87fa-0115f64d8c05 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:53.560899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8460d5a2-6596-4f9b-bff5-fd8702cbd2c2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:53.602091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:53.602675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.602953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.603225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.612706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:53.614523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:53.631176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:53.632283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:53.633335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 382/793] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:33:53 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:53.638485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:53.662707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:53.663361 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.663642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.664212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] PATH_INFO: `/v3/users/swift` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.669634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:53.682441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:53.684146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:53.706501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:53.707620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:53.708542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:53.709067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 412/794] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:53 2022] GET /identity/v3/users/swift => generated 83 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:53.715595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:53.788202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:53.796765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464bac3f-ab7a-4361-b7a9-3e41ff212f62 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:53.833831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8976a5d-e90b-4c79-8ab0-028433ba952b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:53.834762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8976a5d-e90b-4c79-8ab0-028433ba952b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.835147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8976a5d-e90b-4c79-8ab0-028433ba952b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.835529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8976a5d-e90b-4c79-8ab0-028433ba952b None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.836371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8976a5d-e90b-4c79-8ab0-028433ba952b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:53.836786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8976a5d-e90b-4c79-8ab0-028433ba952b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:53.838313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8976a5d-e90b-4c79-8ab0-028433ba952b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.858347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 383/795] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Oct 4 08:33:53 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:53.859872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c03fbf0-c42b-4468-8125-dfe2aa55d349 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.860801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c03fbf0-c42b-4468-8125-dfe2aa55d349 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.861259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c03fbf0-c42b-4468-8125-dfe2aa55d349 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.862981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 384/796] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:53.871193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5661777e-31e9-482c-ada0-18317ffcc76d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.871470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5661777e-31e9-482c-ada0-18317ffcc76d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.871794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5661777e-31e9-482c-ada0-18317ffcc76d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.876830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd82c8-8f4d-46ba-8548-984d858fd07f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:53.892125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5661777e-31e9-482c-ada0-18317ffcc76d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:53.893575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5661777e-31e9-482c-ada0-18317ffcc76d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:53.925023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:53.925719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:53.926046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:53.926324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:53.927088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:53.927391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:53.928865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:53.956642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 413/797] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Oct 4 08:33:53 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:53.957588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.004129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 385/798] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:33:54.005418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5661777e-31e9-482c-ada0-18317ffcc76d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.017690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.068976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4825bc-10f3-4980-8286-5232e5f14843 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.117835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fe24d2-bcf2-4e72-9f34-f89092a1ccf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.118686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fe24d2-bcf2-4e72-9f34-f89092a1ccf1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.119589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fe24d2-bcf2-4e72-9f34-f89092a1ccf1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.119878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fe24d2-bcf2-4e72-9f34-f89092a1ccf1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.126837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.135462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fe24d2-bcf2-4e72-9f34-f89092a1ccf1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.140648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fe24d2-bcf2-4e72-9f34-f89092a1ccf1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.151626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5661777e-31e9-482c-ada0-18317ffcc76d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-f9fe24d2-bcf2-4e72-9f34-f89092a1ccf1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:54.163687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:54.164650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:54.165618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:54.165618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:54.165618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:54.169569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 414/799] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:54.170945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf3d1a3-c7bf-4a82-ba2e-6c74a4a0aa8f None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.171154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf3d1a3-c7bf-4a82-ba2e-6c74a4a0aa8f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.171412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf3d1a3-c7bf-4a82-ba2e-6c74a4a0aa8f None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.173979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 415/800] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:54.174868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf3d1a3-c7bf-4a82-ba2e-6c74a4a0aa8f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.199990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a22322-21db-431e-b2c8-27afd64f0be5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.201566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a22322-21db-431e-b2c8-27afd64f0be5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.202304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a22322-21db-431e-b2c8-27afd64f0be5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.202842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a22322-21db-431e-b2c8-27afd64f0be5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.214387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a22322-21db-431e-b2c8-27afd64f0be5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.216064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a22322-21db-431e-b2c8-27afd64f0be5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-07a22322-21db-431e-b2c8-27afd64f0be5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:54.231404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:54.232334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:54.233207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:54.233207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:54.233207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:54.233207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 386/801] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:54.235639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe44ae12-5638-485c-9787-79b803cb7ce6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.235639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe44ae12-5638-485c-9787-79b803cb7ce6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.235639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe44ae12-5638-485c-9787-79b803cb7ce6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.254002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-fe44ae12-5638-485c-9787-79b803cb7ce6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:54.255603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fe44ae12-5638-485c-9787-79b803cb7ce6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:54.291007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf3d1a3-c7bf-4a82-ba2e-6c74a4a0aa8f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.293483 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.326271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.326495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 387/802] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:33:54.326802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.326908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.327256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.327357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3274fa5e-01ea-4ee4-a5d1-65221ce1b294 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.327651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3274fa5e-01ea-4ee4-a5d1-65221ce1b294 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.327824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:54.328057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3274fa5e-01ea-4ee4-a5d1-65221ce1b294 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.328163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.329486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.334047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 388/803] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:54.334047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3274fa5e-01ea-4ee4-a5d1-65221ce1b294 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.351017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 416/804] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:54.351017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.415177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.457720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747943d1-6a93-4d44-b426-388de4f9973d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.458856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3274fa5e-01ea-4ee4-a5d1-65221ce1b294 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.491410 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.492038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.492311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.492617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.494356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8a13ad-90bf-4ece-8940-d453e0868dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.494998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8a13ad-90bf-4ece-8940-d453e0868dc5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.495591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8a13ad-90bf-4ece-8940-d453e0868dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.495881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8a13ad-90bf-4ece-8940-d453e0868dc5 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.496516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8a13ad-90bf-4ece-8940-d453e0868dc5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:54.497042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8a13ad-90bf-4ece-8940-d453e0868dc5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.499067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8a13ad-90bf-4ece-8940-d453e0868dc5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.503340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:54.508498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:54.509583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:54.510527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:54.517981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 389/805] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:54.518124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 417/806] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:54.518220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c831f9ef-f665-4933-bf35-a024c8855ebf None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.518220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c831f9ef-f665-4933-bf35-a024c8855ebf None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.518220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c831f9ef-f665-4933-bf35-a024c8855ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.518436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.534074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c831f9ef-f665-4933-bf35-a024c8855ebf None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:54.535275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c831f9ef-f665-4933-bf35-a024c8855ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:54.543432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.610673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 390/807] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:33:54.611513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c831f9ef-f665-4933-bf35-a024c8855ebf None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.620225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13340b8-d0ba-4265-9637-88ea5211ee6e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.656923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.658000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.658448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.658864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.663465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.667089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.668734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:54.680183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:54.681054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:54.681915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:54.685906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 418/808] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:54.686886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.714938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c831f9ef-f665-4933-bf35-a024c8855ebf None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.753318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.753917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.754194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.754485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.755310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:54.755888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.757528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.765901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 391/809] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:54.767500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.780945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.796932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6b7dc4-0420-4ac9-bdd0-346466d4dd6f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.833881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.834995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.835367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.835700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.842593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.844006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:54.856359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:54.857294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:54.858248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 419/810] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:54.859363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.878958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929f8ada-d10f-4f04-8312-c492a8900f2a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.892449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:54.912966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.913523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.913795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.914061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] PATH_INFO: `/v3/services/network` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.914663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.916003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:54.926024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:54.927221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:54.928052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 392/811] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/services/network => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:54.928839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:54.961652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b296dd2-be82-4174-a059-b32b73f6ed46 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:54.994070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:54.994694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:54.994967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:54.995233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:54.995906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:54.996208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:54.997611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.009534 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 420/812] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:55.010313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.013598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:55.027258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a81482-6661-4968-9207-9c3fa264e3ea None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.060507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.061133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.061414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.061682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.076407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.077854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:55.092390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:55.093435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:55.095099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:55.095099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:55.095099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:55.095099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 393/813] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:55.095099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.109041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05f7524-5219-4b19-875e-a71165b2ebb6 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.125276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:55.144071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.144745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.145025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.145291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.146031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:55.146263 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.147800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.164371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 421/814] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:55.164371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.196972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a924db-ac12-4432-b8bf-abbc0a6b9b8b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.233665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.234305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.234597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.235510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.236481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.237872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:55.247762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:55.248820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:55.249667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 394/815] 10.222.0.27 () {56 vars in 1129 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:55.251996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.342780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:55.356322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:55.365755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b104b0-1cb7-42fc-8ac0-8fac4352e8bc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.386333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8907a3-0b19-4f96-9560-3d7c7a10f289 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.405063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.405720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.406001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.406274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/c9cff7d749314dbc94645a7884792496/roles/4b9e5859f89942e79cfb0e553769100b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.419623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.420155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.420427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.420692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.420797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=c9cff7d749314dbc94645a7884792496, role_id=4b9e5859f89942e79cfb0e553769100b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.421304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] RBAC: Adding query filter params (name=network) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:55.421596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.422145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.423574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.428112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 422/816] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/services?name=network => generated 124 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:55.428953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.522785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6930e1-ea68-4fb9-92c3-61e827dc1263 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.547639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:55.555195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945658d-6d05-472f-b143-73390057d93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.555790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945658d-6d05-472f-b143-73390057d93d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.556066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945658d-6d05-472f-b143-73390057d93d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.556335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945658d-6d05-472f-b143-73390057d93d None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.556971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5945658d-6d05-472f-b143-73390057d93d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:55.557268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5945658d-6d05-472f-b143-73390057d93d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.558645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5945658d-6d05-472f-b143-73390057d93d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.565061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 395/817] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:55 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/c9cff7d749314dbc94645a7884792496/roles/4b9e5859f89942e79cfb0e553769100b => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:55.568889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.571230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 423/818] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:55.571878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945658d-6d05-472f-b143-73390057d93d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.697913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456b65c6-d492-4155-9fb8-709b91a243f9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.700254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945658d-6d05-472f-b143-73390057d93d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.735395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.736072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.736351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.736618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/085a0cc3f5cb477989e6e75adea7c53a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.738648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=085a0cc3f5cb477989e6e75adea7c53a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.740073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.740520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.741308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.741740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.742182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.743045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:55.743822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.745232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.750221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 424/819] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:55.751226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.809407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:55.826669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 396/820] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:33:55 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/085a0cc3f5cb477989e6e75adea7c53a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:33:55.827670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.834814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:55.871788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad813dc9-31ca-403e-9e48-d358e9cf6d4b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.912784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.913779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.914232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.914714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:55.915551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] RBAC: Adding query filter params (type=network) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:55.916045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:55.917734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:55.929088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 425/821] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/services?type=network => generated 350 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:55.942105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:55.958019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841c1ce6-04c2-4dbb-9dd0-ac70d9b59148 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:55.998100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:55.998790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:55.999066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:55.999353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] PATH_INFO: `/v3/users/nova` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:56.009354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:56.010879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:56.024825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:56.025869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:56.027373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 397/822] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:56.028183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:56.059056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e58ff-2fff-48c3-b85e-98bd3ea8fec5 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:56.082854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:56.102519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:56.103624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:56.104124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:56.104630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:56.105460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:56.105985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:56.107866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:56.113207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:56.114150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 426/823] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Oct 4 08:33:55 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:56.114943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:56.168885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29aa49f-7575-465f-a179-6bd16637462e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:56.210873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:56.211577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:56.211958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:56.212332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:56.213210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:56.214796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:56.229793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac46eda6-cd18-4afd-82aa-bc90e90434cb None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:56.266753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:56.267344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:56.267615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:56.267879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:56.268583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:56.268927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:56.270367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:56.292683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 427/824] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:33:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:56.292683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:56.334904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 398/825] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:33:56 2022] POST /identity/v3/endpoints => generated 337 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:56.336662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:56.337968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:56.462720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ced3b9-2f13-4602-ada7-afbd45ad11df None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:56.501652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a826d5d5-d588-4d26-a847-dd459675602d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:56.522691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:56.537903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:56.538472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:56.538750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:56.539055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:56.539910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:56.541094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:56.545060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:56.546220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:56.546760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:56.547852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:56.633722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:56.635715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:56.698515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 428/826] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:33:56 2022] POST /identity/v3/endpoints => generated 346 bytes in 410 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:33:56.758110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 399/827] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:56 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:33:56.764271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:56.884726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:56.898388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44510af-cef7-4c1a-9a65-383dfae3a579 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:56.935298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:56.936692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:56.936974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:56.937545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:56.948672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:56.950352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:56.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:56.969090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:56.970637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:56.970637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:56.970637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:56.970637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 429/828] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:56.975134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:56.995286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:57.104076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53013aa9-86ea-42e2-9ff3-e47c0eaa7f6e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:57.143308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:57.144288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:57.144795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:57.145268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:57.146131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:57.146688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:57.148265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:57.161626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 400/829] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:57.167507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:57.277670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0e8c3d-0ad9-4493-869c-6f60ad4fd3ff None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:57.314953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:57.315589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:57.315869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:57.316141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:57.323169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:57.324686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:33:57.335846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:57.336837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:57.337707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 430/830] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:33:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:57.343032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:57.437840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:57.461976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c3487e-24c0-4b3a-90de-094a2cab4826 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:57.502995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d261a48-dc11-491e-9902-3a1a2cef1951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:57.503611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d261a48-dc11-491e-9902-3a1a2cef1951 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:57.503889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d261a48-dc11-491e-9902-3a1a2cef1951 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:57.504163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d261a48-dc11-491e-9902-3a1a2cef1951 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:57.504855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d261a48-dc11-491e-9902-3a1a2cef1951 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:57.505159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d261a48-dc11-491e-9902-3a1a2cef1951 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:57.506728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d261a48-dc11-491e-9902-3a1a2cef1951 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:57.520615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 401/831] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:33:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:57.526701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:57.530285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:57.638684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cb4d9d-a643-46a6-b62e-9805d5d239a4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:57.677266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f539067a-10c4-4c5e-a0af-5b7dba0af69e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:57.677876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539067a-10c4-4c5e-a0af-5b7dba0af69e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:57.678161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539067a-10c4-4c5e-a0af-5b7dba0af69e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:57.678475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539067a-10c4-4c5e-a0af-5b7dba0af69e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:57.679214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539067a-10c4-4c5e-a0af-5b7dba0af69e None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=40e12aa547b84ac38fd222a9f5f62ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:57.679520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539067a-10c4-4c5e-a0af-5b7dba0af69e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:57.680946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539067a-10c4-4c5e-a0af-5b7dba0af69e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:57.687425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 431/832] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:33:57 2022] GET /identity/v3/role_assignments?user.id=40e12aa547b84ac38fd222a9f5f62ae2&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:58.070942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff936d8a-2ce1-497a-81fa-8cb1cf2298cc None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.071439 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff936d8a-2ce1-497a-81fa-8cb1cf2298cc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.071752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff936d8a-2ce1-497a-81fa-8cb1cf2298cc None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.073232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 402/833] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:58.078945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b488da14-8d71-486b-91d1-24d144b63c73 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.079793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b488da14-8d71-486b-91d1-24d144b63c73 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.080076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b488da14-8d71-486b-91d1-24d144b63c73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.091899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4cf26-c6a1-460f-91c3-dea805790686 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.092193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4cf26-c6a1-460f-91c3-dea805790686 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.092435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4cf26-c6a1-460f-91c3-dea805790686 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.093862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 403/834] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:58.099613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1d255-6177-4dcc-bab4-0e01f78ebdd3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.099892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1d255-6177-4dcc-bab4-0e01f78ebdd3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.100164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1d255-6177-4dcc-bab4-0e01f78ebdd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.105196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b488da14-8d71-486b-91d1-24d144b63c73 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:58.106683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b488da14-8d71-486b-91d1-24d144b63c73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:58.122277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-95a1d255-6177-4dcc-bab4-0e01f78ebdd3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:58.123558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-95a1d255-6177-4dcc-bab4-0e01f78ebdd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:58.174351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:58.182906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 432/835] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:33:58.212896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 404/836] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:33:58.486622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b488da14-8d71-486b-91d1-24d144b63c73 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:58.511816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a1d255-6177-4dcc-bab4-0e01f78ebdd3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:58.595129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:58.606988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b488da14-8d71-486b-91d1-24d144b63c73 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:58.637250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a1d255-6177-4dcc-bab4-0e01f78ebdd3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:58.654956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:58.656191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.657236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.658225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.673208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:58.679286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:58.680199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:33:58.681022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:58.681663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:58.681663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.681663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.681663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.686471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 433/837] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:58.689372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:58.697821 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:33:58.704952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:58.705892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:33:58.707155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:58.710479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 405/838] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:33:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:58.711912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:58.773624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:58.829310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce2dd2c-e95d-4db4-859f-ca31d60224b0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:58.848484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872d2aa0-0e51-46be-a515-d7e1981937b0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:58.870943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:58.871548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.871830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.872100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.872758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:58.873054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:58.874653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:58.886382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:58.893555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 434/839] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:33:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:58.894899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:58.901825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:58.901825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:58.901825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:58.901825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:58.902154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:58.902307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:58.903727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:58.917870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 406/840] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:33:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:33:58.918726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.019721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1d33ec-729d-4569-a41a-93c7ae75687d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.037334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.049299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5143f86e-457b-485b-9b04-584a22781299 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.055291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3ece33-c85e-429c-b387-8e527bb2f59a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.055940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3ece33-c85e-429c-b387-8e527bb2f59a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.056216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3ece33-c85e-429c-b387-8e527bb2f59a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.056486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3ece33-c85e-429c-b387-8e527bb2f59a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.070204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3ece33-c85e-429c-b387-8e527bb2f59a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.071692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3ece33-c85e-429c-b387-8e527bb2f59a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-5e3ece33-c85e-429c-b387-8e527bb2f59a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:59.090746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:59.092065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:59.092908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:59.092908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:59.092908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:59.092908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 435/841] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:59.100631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f34d90-00b1-4784-ab0a-a2a447f61159 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.100909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f34d90-00b1-4784-ab0a-a2a447f61159 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.101176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f34d90-00b1-4784-ab0a-a2a447f61159 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.102601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 436/842] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:33:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:59.103782 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f34d90-00b1-4784-ab0a-a2a447f61159 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.103896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0011cee7-322f-4251-9159-72be6ae34af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.104309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0011cee7-322f-4251-9159-72be6ae34af4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.104588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0011cee7-322f-4251-9159-72be6ae34af4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.104851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0011cee7-322f-4251-9159-72be6ae34af4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.118651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0011cee7-322f-4251-9159-72be6ae34af4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.120093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0011cee7-322f-4251-9159-72be6ae34af4 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-0011cee7-322f-4251-9159-72be6ae34af4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:33:59.136448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:59.137440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:33:59.138311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:33:59.138311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:33:59.138311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:59.138311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 407/843] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:33:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:59.139026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7a95b-130f-4fdc-bef9-8987265e9e71 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.139026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7a95b-130f-4fdc-bef9-8987265e9e71 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.139198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7a95b-130f-4fdc-bef9-8987265e9e71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.158658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5ab7a95b-130f-4fdc-bef9-8987265e9e71 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:59.159839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5ab7a95b-130f-4fdc-bef9-8987265e9e71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:59.194661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.215461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f34d90-00b1-4784-ab0a-a2a447f61159 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.228974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 408/844] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:33:59 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:33:59.229809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab7a95b-130f-4fdc-bef9-8987265e9e71 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.264103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.265195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.265615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.266658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.267438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:59.268133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.270554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.285673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 437/845] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:59.292060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.330978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.343883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab7a95b-130f-4fdc-bef9-8987265e9e71 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.397446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.398731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.399229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.399718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.400862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:59.401342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.403512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.421270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 409/846] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:59.422735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.430445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62e7e19-b6c5-432b-9201-fa679a0bc6fe None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.458894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.470049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.470981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.473818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.473818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] PATH_INFO: `/v3/users/swift` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.479623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.481328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:59.493881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:59.495030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:33:59.496116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:33:59.498183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 438/847] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/users/swift => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:59.505662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.535920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7b51ab-b6ba-40b7-8c60-0acce4730e4c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.573055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.573675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.573990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.574264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.582354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.584251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:33:59.596002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:59.599061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 410/848] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:59.599984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.609758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.619334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93509e8-3bf2-4893-ad2a-77aaebbe26dd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.653911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.655480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.655480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.655480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.655954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:59.656245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.657712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.676849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 439/849] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:59.682162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.699926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.710510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1905953b-556e-4a7f-97ea-d2e3c0b3832b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.746627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.747410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.747690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.747962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.748548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.749910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:33:59.761957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:33:59.763561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:33:59.764773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 411/850] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:33:59.765583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.797563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5edbeb-4e9c-4f90-90ae-957170ae5dd2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.834513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.835121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.835400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.835667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.836380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:33:59.836679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.838213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:59.863140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 440/851] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:59.863140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:59.874336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.882936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ad7edb-3bc1-4b59-9acb-22bfa5c71ecf None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.920319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df157c7d-a95c-4859-ab39-ff7680e237b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:59.920927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df157c7d-a95c-4859-ab39-ff7680e237b5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:59.921205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df157c7d-a95c-4859-ab39-ff7680e237b5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:59.921474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df157c7d-a95c-4859-ab39-ff7680e237b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:59.949400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:59.971283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86f4e6-953b-42aa-a281-e1ae490c749a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:59.997851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df157c7d-a95c-4859-ab39-ff7680e237b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:59.999719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df157c7d-a95c-4859-ab39-ff7680e237b5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.005576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.006248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.006616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.006886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.007500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:00.007796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.009126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.013420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 441/852] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:00.014179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.082226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.101314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 412/853] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:33:59 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:00.101314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d02b51-d9ae-454e-a701-3a7246cb39d1 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.101314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d02b51-d9ae-454e-a701-3a7246cb39d1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.101314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d02b51-d9ae-454e-a701-3a7246cb39d1 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.101314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 413/854] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:00.101314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d02b51-d9ae-454e-a701-3a7246cb39d1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.113138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c96874-f104-4e8d-852d-8feb57bcb6db None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.148238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a25a9-33f0-4b28-bdf4-68b1588fa60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.148844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a25a9-33f0-4b28-bdf4-68b1588fa60b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.149119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a25a9-33f0-4b28-bdf4-68b1588fa60b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.149386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a25a9-33f0-4b28-bdf4-68b1588fa60b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.161235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.206973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d02b51-d9ae-454e-a701-3a7246cb39d1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.216325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a25a9-33f0-4b28-bdf4-68b1588fa60b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.217767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a25a9-33f0-4b28-bdf4-68b1588fa60b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.239039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.239639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.239921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.240191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.240808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:00.241106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.242466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.246951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 414/855] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:00.247709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.289014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.323970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 442/856] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:00.325639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0edffd5-56c4-46b7-8b0e-e549c157d590 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.326152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0edffd5-56c4-46b7-8b0e-e549c157d590 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.326673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0edffd5-56c4-46b7-8b0e-e549c157d590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.346384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f0edffd5-56c4-46b7-8b0e-e549c157d590 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:00.348049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f0edffd5-56c4-46b7-8b0e-e549c157d590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:00.376196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405479b2-df34-4083-8300-7a95829b3de9 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.401956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.422756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.423459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.424368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.424368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.435553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 443/857] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:00 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:00.436456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0edffd5-56c4-46b7-8b0e-e549c157d590 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.436652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.439571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:00.457489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:00.458492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:00.459372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:00.459372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:00.459372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:00.461729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 415/858] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:00.468235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.540402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.544362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0edffd5-56c4-46b7-8b0e-e549c157d590 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.581287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.582266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.582847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.583319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.596200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.597645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.610237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e6e596-942d-4848-984f-6e18d4470261 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:00.620860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:00.622094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:00.623028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:00.623028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:00.623028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:00.624692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 444/859] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:00.634067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.651665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.652361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.652713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.653391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.653741 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:00.654126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.655575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.668930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 416/860] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:00.675028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.707987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.753644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6eebc8-b78f-42a1-be1d-ccacd2279b56 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.787437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.788040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.788314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.788582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.788690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5126361-ba04-411b-8aea-2c7c5c1a17bb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.789219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:00.789517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.790983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.808135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 445/861] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:00.808962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.827994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.829174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.829664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.830258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.832974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.838380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.839911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:00.852676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:00.853592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 417/862] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:00.855104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:00.926648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40524733-e3e5-41e1-8d61-eea06d34aa23 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.957339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:00.963372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:00.964014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:00.964291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:00.964563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:00.965142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:00.966528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:00.968094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4708c3-a476-4972-bce9-8d55e134d86f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:00.976653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:00.983400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:00.984434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 446/863] 10.222.0.27 () {56 vars in 1132 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:00.986279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:01.014138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:01.014815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:01.015188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:01.015355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:01.023774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:01.025637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:01.039066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:01.040159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:01.041292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:01.042789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 418/864] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:01.043466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:01.067624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:01.095499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1801a70a-41a6-4442-92dc-1e8e4e52e6d9 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:01.139078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:01.139696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:01.139975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:01.140248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:01.141599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:01.142060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:01.143616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:01.155585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5b0088-ce4c-4d4e-93af-f4b349f818f1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:01.161699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 447/865] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:01.161699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:01.194247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:01.194878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:01.195151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:01.195531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:01.196268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:01.196561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:01.197908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:01.206979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 419/866] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:01 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:01.206979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:01.221536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:01.266455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1178d68a-2dd8-415d-a2fe-aa5e54ceb9e4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:01.303222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:01.303742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:01.304020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:01.304288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:01.304965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:01.305475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:01.307119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:01.310088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:01.316739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b3060c-c05f-4340-a467-9dd8177c2c48 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:01.320809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 448/867] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:01.321092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:01.357483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:01.358128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:01.358417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:01.358694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:01.359406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=085a0cc3f5cb477989e6e75adea7c53a) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:01.360237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:01.361644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:01.378436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 420/868] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:01 2022] GET /identity/v3/role_assignments?user.id=085a0cc3f5cb477989e6e75adea7c53a&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:01.378436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:01.447051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdcdeac-cec0-433c-98af-cffb37413890 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:01.466024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:01.490093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4736d59f-5e0c-46c7-966f-5b812f419f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:01.490766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736d59f-5e0c-46c7-966f-5b812f419f9a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:01.491042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736d59f-5e0c-46c7-966f-5b812f419f9a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:01.491311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736d59f-5e0c-46c7-966f-5b812f419f9a None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:01.502823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4736d59f-5e0c-46c7-966f-5b812f419f9a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:01.502973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434b8e8-364b-42af-b15a-a9009c6bed9f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:01.503080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4736d59f-5e0c-46c7-966f-5b812f419f9a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:01.503080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4736d59f-5e0c-46c7-966f-5b812f419f9a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:01.503080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 449/869] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:01 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:01.536721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf52630c-8bb2-45c3-b7ac-801ee16ddd59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:01.537363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf52630c-8bb2-45c3-b7ac-801ee16ddd59 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:01.537644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf52630c-8bb2-45c3-b7ac-801ee16ddd59 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:01.537918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf52630c-8bb2-45c3-b7ac-801ee16ddd59 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:01.538742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf52630c-8bb2-45c3-b7ac-801ee16ddd59 None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=c9cff7d749314dbc94645a7884792496) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:01.539056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf52630c-8bb2-45c3-b7ac-801ee16ddd59 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:01.540663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf52630c-8bb2-45c3-b7ac-801ee16ddd59 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:01.548722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 421/870] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:01 2022] GET /identity/v3/role_assignments?user.id=c9cff7d749314dbc94645a7884792496&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:02.471848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa6e411-af5a-41b7-aea9-e78216c76099 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:02.472419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa6e411-af5a-41b7-aea9-e78216c76099 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:02.472419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa6e411-af5a-41b7-aea9-e78216c76099 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:02.473994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 450/871] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:02.479146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e14656-c738-43a5-81d4-0439c7885f29 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:02.479541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e14656-c738-43a5-81d4-0439c7885f29 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:02.479681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e14656-c738-43a5-81d4-0439c7885f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:02.498028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c7e14656-c738-43a5-81d4-0439c7885f29 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:02.499227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c7e14656-c738-43a5-81d4-0439c7885f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:02.575698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 422/872] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:02 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:02.848049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa6e411-af5a-41b7-aea9-e78216c76099 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:02.920157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:02.963546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa6e411-af5a-41b7-aea9-e78216c76099 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:03.001658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13244948-f0d6-4c51-8933-38c4d7fa481e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:03.002259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13244948-f0d6-4c51-8933-38c4d7fa481e None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:03.002560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13244948-f0d6-4c51-8933-38c4d7fa481e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:03.002829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13244948-f0d6-4c51-8933-38c4d7fa481e None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:03.003474 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13244948-f0d6-4c51-8933-38c4d7fa481e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:03.004877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13244948-f0d6-4c51-8933-38c4d7fa481e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:03.085270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 451/873] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:34:02 2022] POST /identity/v3/services => generated 251 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:03.705335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2274ad-d362-43cc-b14d-76ca93372ad8 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:03.706353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2274ad-d362-43cc-b14d-76ca93372ad8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:03.706866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2274ad-d362-43cc-b14d-76ca93372ad8 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:03.708641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 423/874] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:03.714134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f377e8-8970-4153-b1d4-9b1f6c9191d2 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:03.714959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f377e8-8970-4153-b1d4-9b1f6c9191d2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:03.715446 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f377e8-8970-4153-b1d4-9b1f6c9191d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:03.723570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04282aa-2446-438f-a3ac-e36ed8f43e17 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:03.723854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04282aa-2446-438f-a3ac-e36ed8f43e17 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:03.724127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04282aa-2446-438f-a3ac-e36ed8f43e17 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:03.725537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 424/875] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:03.730418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1640422-c8cf-4ae4-8acc-9bcac0440a64 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:03.730708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1640422-c8cf-4ae4-8acc-9bcac0440a64 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:03.730980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1640422-c8cf-4ae4-8acc-9bcac0440a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:03.737602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d6f377e8-8970-4153-b1d4-9b1f6c9191d2 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:03.739223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d6f377e8-8970-4153-b1d4-9b1f6c9191d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:03.751326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b1640422-c8cf-4ae4-8acc-9bcac0440a64 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:03.752582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b1640422-c8cf-4ae4-8acc-9bcac0440a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:03.825638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:03.833947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 425/876] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:03 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:03.840436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 452/877] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:03 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:03.874882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe5b770-3889-40bb-9ee6-3cc6f66bbc17 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:03.875950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe5b770-3889-40bb-9ee6-3cc6f66bbc17 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:03.875950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe5b770-3889-40bb-9ee6-3cc6f66bbc17 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:03.877543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 426/878] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:03.882853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127adcc-a5cf-408e-aa06-ea0c60e4a47b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:03.883366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127adcc-a5cf-408e-aa06-ea0c60e4a47b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:03.883835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127adcc-a5cf-408e-aa06-ea0c60e4a47b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:03.903812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6127adcc-a5cf-408e-aa06-ea0c60e4a47b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:03.905300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6127adcc-a5cf-408e-aa06-ea0c60e4a47b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:03.951140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:03.977377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 453/879] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:03 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:04.102780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe5b770-3889-40bb-9ee6-3cc6f66bbc17 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:04.134676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6127adcc-a5cf-408e-aa06-ea0c60e4a47b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:04.220785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe5b770-3889-40bb-9ee6-3cc6f66bbc17 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:04.244788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6127adcc-a5cf-408e-aa06-ea0c60e4a47b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:04.265233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:04.266102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:04.266356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:04.266735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:04.267468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:04.268891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:04.280980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:04.281642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:04.281959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:04.282185 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:04.282792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:04.284136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:04.292890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:04.294170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:04.295271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:04.298563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 454/880] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:04 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:04.299539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:04.340818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:04.382422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 427/881] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:34:04 2022] POST /identity/v3/services => generated 269 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:04.385294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:04.408821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b935ef06-c684-4ca2-8773-ab89d3c0e04b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:04.456379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:04.457084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:04.457358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:04.457657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:04.470440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:04.476236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:04.477245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:04.478062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 455/882] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Oct 4 08:34:04 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:04.481921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:04.483199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:04.534103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6b02d4-cc90-4199-b1e8-593c42db29d0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:04.574328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:04.575169 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:04.576345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:04.576716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:04.577450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:04.577853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:04.579449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:04.586461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 428/883] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:04 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:04.591098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:04.618120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:04.621885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d046c1-c66a-4dc3-8e15-2e070a397c09 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:04.660720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:04.661336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:04.661613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:04.661881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:04.662586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:04.663247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:04.664475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:04.674328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 456/884] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Oct 4 08:34:04 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:04.680057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:04.720883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9351e4-428a-41b8-a7be-8474d988dea1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:04.750618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:04.764959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:04.766426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:04.766850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:04.767327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:04.768163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:04.768641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:04.770655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:04.782481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 429/885] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:04 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:04.804515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77fd36a-e4dc-4291-8cee-c0d021c97cf3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:04.846359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:04.847106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:04.847390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:04.847661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:04.859571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:04.861146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:04.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:04.879614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:04.880476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:04.880476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:04.880476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:04.880476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 457/886] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:04.885779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:05.008345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d62a98a-84bb-414b-a2bc-41d61cf2c76d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:05.044715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:05.045649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.046077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.046524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.047861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:05.048329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:05.049838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:05.063016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 430/887] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:05.068274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:05.087995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:05.198844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedd4b33-fe42-4424-9f15-73654b91d903 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:05.239618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb6b8aa-d7b5-4ca0-924c-c2c8fb7e3feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:05.240281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6b8aa-d7b5-4ca0-924c-c2c8fb7e3feb None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.240566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6b8aa-d7b5-4ca0-924c-c2c8fb7e3feb None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.240842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6b8aa-d7b5-4ca0-924c-c2c8fb7e3feb None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.250080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb6b8aa-d7b5-4ca0-924c-c2c8fb7e3feb None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:05.251839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb6b8aa-d7b5-4ca0-924c-c2c8fb7e3feb None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-1fb6b8aa-d7b5-4ca0-924c-c2c8fb7e3feb None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:34:05.264315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:05.265221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:34:05.266719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:05.267758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 458/888] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:34:05 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:05.273592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:05.325193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:05.339498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20f63ca-8d0b-44c5-b081-b2a56e922002 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.345865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20f63ca-8d0b-44c5-b081-b2a56e922002 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.345865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20f63ca-8d0b-44c5-b081-b2a56e922002 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.345865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 459/889] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:05.346960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e590ed46-e5c9-47f3-84cb-b3e4f5d37c1f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.347244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e590ed46-e5c9-47f3-84cb-b3e4f5d37c1f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.347517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e590ed46-e5c9-47f3-84cb-b3e4f5d37c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.366826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e590ed46-e5c9-47f3-84cb-b3e4f5d37c1f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:05.368048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e590ed46-e5c9-47f3-84cb-b3e4f5d37c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:05.393643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0446fb-ae91-4413-8acc-4f9b6d9fe6a1 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:05.432507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:05.433225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.433573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.433911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.434899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:05.435273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:05.436905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:05.454384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:05.464353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 460/891] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:05 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:05.464531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 431/891] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Oct 4 08:34:05 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:05.466009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:05.573865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2471b6a-03f5-4d72-9cfc-ce0371f82996 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:05.576693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:05.610649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:05.611518 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.611947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.612385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.687584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:05.689937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:05.779206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e590ed46-e5c9-47f3-84cb-b3e4f5d37c1f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:05.802160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 432/892] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:05 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:05.814448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:05.834868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:05.902663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e590ed46-e5c9-47f3-84cb-b3e4f5d37c1f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:05.935420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaec508-3169-491f-9ac7-09b9db81a16a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:05.945350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:05.946088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.946372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.946673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.947255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:05.948738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:05.958303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:05.959299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:05.960130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 461/893] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:34:05 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:05.967358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:05.975802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:05.976542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:05.976818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:05.977084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:05.988427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:05.989851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:06.006437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:06.008002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:06.009098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:06.009098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:06.009098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:06.009098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 433/894] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:06.013207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:06.057947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:06.080321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:06.110362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917fa009-afca-4e9c-a72f-33b7867b1217 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.128552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3151fb4-86ef-41f7-9d07-fb9145df3be8 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.153929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.155142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.155622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.156851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.157990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:06.158465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.160713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.164861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.165655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.166042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.166427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.166898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 462/895] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:34:05 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:06.167638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:06.168663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.174248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.176053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:06.192826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 434/896] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:06.193871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:06.291676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1824c64b-8876-4f3e-b501-512e119b1d3c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.319135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:06.320066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2350f102-7bbe-4eeb-ae8d-3f35be452f46 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.327714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.328422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.328762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:06.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 463/897] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:34:06 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:06.342329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:06.348112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:06.370127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.371276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.372028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.372869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.382745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.384932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:06.400453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:06.401354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:06.402231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:06.409566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 435/898] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:06.411667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:06.473522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e814472-078d-4c3f-922f-52d352a75229 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.517676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d180b5-8556-4d78-a847-d91bc3ea9386 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.518283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.519303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.519808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.521075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.521994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] RBAC: Adding query filter params (interface=public, service_id=d8e55c19c5f944399dd9ecb818d0d373, region_id=RegionOne) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:06.522552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.524138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.535889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 464/899] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:34:06 2022] GET /identity/v3/endpoints?service_id=d8e55c19c5f944399dd9ecb818d0d373&interface=public®ion_id=RegionOne => generated 194 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:06.536738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:06.566941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.568673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.569160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.569907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.571563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:06.572109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.574966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.596425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 436/900] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:06.597996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:06.601560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:06.648389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:06.651207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50372d9b-bf20-4716-92c4-21d3cef205d8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.697775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd160fc8-11b1-45e0-98bd-507f369ac5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.698567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd160fc8-11b1-45e0-98bd-507f369ac5aa None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.698957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd160fc8-11b1-45e0-98bd-507f369ac5aa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.699328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd160fc8-11b1-45e0-98bd-507f369ac5aa None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.700032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd160fc8-11b1-45e0-98bd-507f369ac5aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.701488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd160fc8-11b1-45e0-98bd-507f369ac5aa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.714486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 465/901] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:34:06 2022] GET /identity/v3/services => generated 1268 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:34:06.743953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24c6f49-b2bb-4655-99c0-b50fa69afb69 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:06.781107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54f08e-8582-46fd-90cc-1a09030a94dd None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.781961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54f08e-8582-46fd-90cc-1a09030a94dd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.782869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54f08e-8582-46fd-90cc-1a09030a94dd None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.784086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 466/902] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:06.788190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cff910-fcaa-4f2a-a362-54f12aa9d71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:06.788878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cff910-fcaa-4f2a-a362-54f12aa9d71e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.789203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cff910-fcaa-4f2a-a362-54f12aa9d71e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.790517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cff910-fcaa-4f2a-a362-54f12aa9d71e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.790858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846b9d12-2016-4cca-b46e-19fc68320404 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:06.791129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846b9d12-2016-4cca-b46e-19fc68320404 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:06.791394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846b9d12-2016-4cca-b46e-19fc68320404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:06.792228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cff910-fcaa-4f2a-a362-54f12aa9d71e None admin] RBAC: Adding query filter params (role.id=96eb47d9c9e341ab91ecffdb8ea813c8, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=c9cff7d749314dbc94645a7884792496) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:06.792775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cff910-fcaa-4f2a-a362-54f12aa9d71e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:06.794481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cff910-fcaa-4f2a-a362-54f12aa9d71e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:06.802134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 437/903] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:06 2022] GET /identity/v3/role_assignments?user.id=c9cff7d749314dbc94645a7884792496&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=96eb47d9c9e341ab91ecffdb8ea813c8 => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:06.812043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-846b9d12-2016-4cca-b46e-19fc68320404 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:06.814165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-846b9d12-2016-4cca-b46e-19fc68320404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:06.894679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 467/904] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:06 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:07.155104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9fc3cf-4a3d-4d82-b9c9-fa02a6f50d9f None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:07.155626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9fc3cf-4a3d-4d82-b9c9-fa02a6f50d9f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:07.155626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9fc3cf-4a3d-4d82-b9c9-fa02a6f50d9f None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:07.157379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 438/905] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:07.162975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201dd55b-6416-4eb6-8701-dd25a6ce101c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:07.163408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201dd55b-6416-4eb6-8701-dd25a6ce101c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:07.163604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201dd55b-6416-4eb6-8701-dd25a6ce101c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:07.183124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9fc3cf-4a3d-4d82-b9c9-fa02a6f50d9f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:07.191638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-201dd55b-6416-4eb6-8701-dd25a6ce101c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:07.192837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-201dd55b-6416-4eb6-8701-dd25a6ce101c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:07.241156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:07.263309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 468/906] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:07 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:07.301018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:07.305266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9fc3cf-4a3d-4d82-b9c9-fa02a6f50d9f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:07.341676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:07.342298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:07.342645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:07.342916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:07.343494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:07.345460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:07.354561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:07.355464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:07.356335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 439/907] 10.222.0.27 () {56 vars in 1132 bytes} [Tue Oct 4 08:34:07 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:07.363587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201dd55b-6416-4eb6-8701-dd25a6ce101c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:07.465396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201dd55b-6416-4eb6-8701-dd25a6ce101c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:07.500545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:07.501236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:07.501508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:07.501772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:07.502397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:07.502714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:07.504078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:07.509155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 469/908] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:07 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:07.514507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:07.555469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:07.605115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:07.629565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a8dd48-6f72-4da0-9ad7-6d5880f76ce6 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:07.666570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003eea0-f2c9-4876-aa13-fa7a6106da65 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:07.668880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:07.670373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:07.670774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:07.671140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:07.671943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:07.672836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:07.674359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:07.681233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 440/909] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:07 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:07.686556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:07.721124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:07.721636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:07.722501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:07.722958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:07.723405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:07.724254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:07.725856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:07.797530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfd0ec9-4d73-4e74-a385-e68aaf6c866c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:07.818552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 470/910] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:34:07 2022] POST /identity/v3/services => generated 241 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:07.851431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9794146c-e6fd-44ab-971f-ddf5c19ae440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:07.852320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9794146c-e6fd-44ab-971f-ddf5c19ae440 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:07.852826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9794146c-e6fd-44ab-971f-ddf5c19ae440 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:07.853257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9794146c-e6fd-44ab-971f-ddf5c19ae440 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:07.854171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9794146c-e6fd-44ab-971f-ddf5c19ae440 None admin] RBAC: Adding query filter params (interface=public, service_id=26d149745e6e494eaa7cf6c11ac2f42b, region_id=RegionOne) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:07.854646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9794146c-e6fd-44ab-971f-ddf5c19ae440 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:07.856241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9794146c-e6fd-44ab-971f-ddf5c19ae440 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:07.863187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 441/911] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:34:07 2022] GET /identity/v3/endpoints?service_id=26d149745e6e494eaa7cf6c11ac2f42b&interface=public®ion_id=RegionOne => generated 194 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:07.869458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:07.983511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f3072-fa32-4ce5-a97d-6dccecbfc334 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:07.987871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:08.023218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d1a1d0-6194-497b-8b21-0c5edec05306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:08.023873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d1a1d0-6194-497b-8b21-0c5edec05306 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:08.024158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d1a1d0-6194-497b-8b21-0c5edec05306 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:08.024430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d1a1d0-6194-497b-8b21-0c5edec05306 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:08.025037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d1a1d0-6194-497b-8b21-0c5edec05306 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:08.026429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d1a1d0-6194-497b-8b21-0c5edec05306 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:08.033092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 471/912] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:34:07 2022] GET /identity/v3/services => generated 1497 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:34:09.020532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e939ae18-05de-4e13-942f-7f55b8fc8cc2 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.021047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e939ae18-05de-4e13-942f-7f55b8fc8cc2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.021047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e939ae18-05de-4e13-942f-7f55b8fc8cc2 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.022773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 442/913] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:09.027988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce157444-c583-4507-8edd-b8930c8f37b7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.028417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce157444-c583-4507-8edd-b8930c8f37b7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.028594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce157444-c583-4507-8edd-b8930c8f37b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.047325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ce157444-c583-4507-8edd-b8930c8f37b7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:09.048846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ce157444-c583-4507-8edd-b8930c8f37b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:09.091348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fdeea-ad6a-4b8c-afc3-1d28085e4373 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.091992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fdeea-ad6a-4b8c-afc3-1d28085e4373 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.092475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fdeea-ad6a-4b8c-afc3-1d28085e4373 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.094451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 443/914] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:09.099766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49c7b7-56b9-429d-b518-3b693c8df1bc None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.100332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49c7b7-56b9-429d-b518-3b693c8df1bc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.100811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49c7b7-56b9-429d-b518-3b693c8df1bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.119056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 472/915] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:09.119590 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0b49c7b7-56b9-429d-b518-3b693c8df1bc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:09.120905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0b49c7b7-56b9-429d-b518-3b693c8df1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:09.191279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 444/916] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:09.393006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce157444-c583-4507-8edd-b8930c8f37b7 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:09.486279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49c7b7-56b9-429d-b518-3b693c8df1bc None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:09.495165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce157444-c583-4507-8edd-b8930c8f37b7 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:09.504090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:09.510617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:09.534793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:09.535737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.536203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.536654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.537507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:09.539091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:09.548954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:09.549950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:34:09.550790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:09.553508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 473/917] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:34:09 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:09.560121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:09.601854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49c7b7-56b9-429d-b518-3b693c8df1bc None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:09.647387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:09.648633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.649135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.649607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.661587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:09.667366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:09.668300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:09.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:09.672477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 445/918] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Oct 4 08:34:09 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:09.679170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:09.699907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d01f4a7-30ac-4d64-a292-2f794e73a93d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:09.745484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:09.746620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.746908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.747181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.748005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:09.748447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:09.750650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:09.758487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 474/919] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:34:09 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:09.762662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:09.788717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:09.794133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:09.810843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb4ee2f-276a-4b87-8512-5b1d7daecdc5 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:09.849387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:09.850023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.850300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.850601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.851263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:09.851578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:09.852911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:09.866491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 446/920] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Oct 4 08:34:09 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:09.868861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:09.880686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affe3933-a5ad-44cb-85fb-af8aef046893 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:09.920006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:09.920735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:09.921131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:09.921506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:09.922350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:09.922787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:09.924356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:09.930432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 475/921] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Oct 4 08:34:09 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:09.936810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.002186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff680fb4-b661-4752-9485-627f6e9afba2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.044098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:10.047229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.047819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.048149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.048422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.058023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036a371b-3f20-4e1d-9128-18fb390b8afa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.060915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:10.064195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.065967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:10.082790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:10.083840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:10.084743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:10.084743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:10.084743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.092360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 447/922] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:10.093200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.105339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac4df2-5fd7-4141-8b39-c9b0726cd96f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.105953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac4df2-5fd7-4141-8b39-c9b0726cd96f None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.110780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac4df2-5fd7-4141-8b39-c9b0726cd96f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.110780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac4df2-5fd7-4141-8b39-c9b0726cd96f None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.110780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac4df2-5fd7-4141-8b39-c9b0726cd96f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.110780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac4df2-5fd7-4141-8b39-c9b0726cd96f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.194286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 476/923] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:34:09 2022] POST /identity/v3/endpoints => generated 346 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:34:10.195607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a1b146-d1b6-4eea-ba26-485bc85c75dc None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.195959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a1b146-d1b6-4eea-ba26-485bc85c75dc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.196235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a1b146-d1b6-4eea-ba26-485bc85c75dc None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.197573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 477/924] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:10.204120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30612d-9910-47c8-b880-0ef871dcc560 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.204419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30612d-9910-47c8-b880-0ef871dcc560 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.204672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30612d-9910-47c8-b880-0ef871dcc560 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.218010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf9bee-ba86-4b9e-b6d8-1c6c7a553c22 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.226771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9a30612d-9910-47c8-b880-0ef871dcc560 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:10.228142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9a30612d-9910-47c8-b880-0ef871dcc560 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:10.283067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb963d6-4d7e-4fd7-8cd3-bc42fcde1154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.283942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb963d6-4d7e-4fd7-8cd3-bc42fcde1154 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.284203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb963d6-4d7e-4fd7-8cd3-bc42fcde1154 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.284474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb963d6-4d7e-4fd7-8cd3-bc42fcde1154 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.285141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb963d6-4d7e-4fd7-8cd3-bc42fcde1154 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:10.285446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb963d6-4d7e-4fd7-8cd3-bc42fcde1154 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.286880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb963d6-4d7e-4fd7-8cd3-bc42fcde1154 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.302133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 448/925] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:10.303836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd683f28-861b-4df3-ba6a-56a5001b8609 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.304229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd683f28-861b-4df3-ba6a-56a5001b8609 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.304506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd683f28-861b-4df3-ba6a-56a5001b8609 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.305977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 449/926] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:10.311491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd683f28-861b-4df3-ba6a-56a5001b8609 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.317705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 478/927] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:10.320975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f38a68e-94a0-415d-9a33-406da224143e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.320975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f38a68e-94a0-415d-9a33-406da224143e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.320975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f38a68e-94a0-415d-9a33-406da224143e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.343980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8f38a68e-94a0-415d-9a33-406da224143e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:10.345796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8f38a68e-94a0-415d-9a33-406da224143e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:10.367863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:10.434678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd683f28-861b-4df3-ba6a-56a5001b8609 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.441849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 479/928] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:10.476175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.476731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.477004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.477270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.487514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.489013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:34:10.501476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:10.502458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.503771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 450/929] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:10.508304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f38a68e-94a0-415d-9a33-406da224143e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.518108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:10.607076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.618079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f38a68e-94a0-415d-9a33-406da224143e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.660971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.661580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.661856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.662120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.662871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:10.663166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.664689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.684911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 480/930] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:10.695398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.729067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2074c1a-6a96-4f8b-be5b-0699a557a49f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.739575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:10.760249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:10.766729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.767364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.767639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.767908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.768502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.769855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:10.778809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:10.779802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.780613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 451/931] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:10.786462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.803408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9edf971-5dfb-4e4a-bdd3-e2e6d417330d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.840834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.841468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.841743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.842010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.854157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.855744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:10.872150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:10.873123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:10.874002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:10.874002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:10.874002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:10.874002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 481/932] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:10.874218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.898059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc195f3-edf8-47a7-b3c2-0ee3fe4b5bdc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.936022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:10.936655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:10.936950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:10.937204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:10.937840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:10.939212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:10.949787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:10.950982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:34:10.952170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:10.954560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 452/933] 10.222.0.27 () {56 vars in 1132 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:10.955370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:10.983642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad9d74-0799-40a8-b97b-78d6bcadbc86 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:10.997998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.003855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.024318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.043832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.043832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.043832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.043832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:11.043832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.043832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.044690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 482/934] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:11.044690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.071167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156c695e-f7df-440b-8711-2d93751acf0b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.109171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.110035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.110316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.110615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.111260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:11.111685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.113161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.130458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 453/935] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:11.131242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.134389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee46d85e-5426-429a-88a2-868ffed3d3ec None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.167612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.168215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.168489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.168754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.169369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:11.169665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.171048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.178441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 483/936] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:11 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:11.178441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.222296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.238112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.244519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f38e61-d6e6-40bb-8a43-1e27dc3ce993 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.286622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.287236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.287516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.287786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.288162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a0540b-2587-4359-b57a-3df88c4b5ce1 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.288414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:11.288718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.290266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.302216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 454/937] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:11 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:11.302216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.329769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.330398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.330711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.330974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.337608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.339069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:11.350657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:11.351540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:11.352464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 484/938] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:11.352942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.419569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77020ebb-a2b3-477c-94bd-26cef8b80bcb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.457161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f24d1-f3d1-4ada-b0b5-7a7c20371480 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.459635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.461347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.461956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.462231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.462526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.463173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:11.463470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.465300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.471919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 455/939] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:11 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:11.472715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.498002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.498574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.498854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.499133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.499895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] RBAC: Adding query filter params (interface=public, service_id=a89fd48941f5433a85abac6f65759ee7, region_id=RegionOne) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:11.500845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.502652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.503598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.512463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 485/940] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:34:11 2022] GET /identity/v3/endpoints?service_id=a89fd48941f5433a85abac6f65759ee7&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:11.512463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.606933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c41990-9a63-4c5c-8704-a21bb5033f01 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.615054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0733e55-cd96-4bf4-a575-e263b632aa0e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.654659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.655666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.656205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.656904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.657848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:11.658184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.658572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.659107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.659524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.659928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.660776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.660932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.662867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.675594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 456/941] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:11.676445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.763658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.776312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 486/942] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:34:11 2022] POST /identity/v3/endpoints => generated 347 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:11.777368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:11.791484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9265d8dc-a192-41a2-a309-e276a0580aeb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.806859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:11.850877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1372237-ac78-411d-b49c-700a6d723bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.851900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1372237-ac78-411d-b49c-700a6d723bb1 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.852544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1372237-ac78-411d-b49c-700a6d723bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.853034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1372237-ac78-411d-b49c-700a6d723bb1 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.854602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1372237-ac78-411d-b49c-700a6d723bb1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.856617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1372237-ac78-411d-b49c-700a6d723bb1 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:11.866502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 457/943] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:34:11 2022] GET /identity/v3/services => generated 1497 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:34:11.901668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8170576b-3cf6-4a6c-abd1-c0e7c1ee37bc None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:11.943689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7a9b6b-a233-466e-a449-e57db21c5558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:11.944477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a9b6b-a233-466e-a449-e57db21c5558 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:11.944767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a9b6b-a233-466e-a449-e57db21c5558 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:11.945045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a9b6b-a233-466e-a449-e57db21c5558 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/c9cff7d749314dbc94645a7884792496/roles/96eb47d9c9e341ab91ecffdb8ea813c8` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:11.947753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7a9b6b-a233-466e-a449-e57db21c5558 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=c9cff7d749314dbc94645a7884792496, role_id=96eb47d9c9e341ab91ecffdb8ea813c8)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:11.949282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7a9b6b-a233-466e-a449-e57db21c5558 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:12.035206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 487/944] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:34:11 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/c9cff7d749314dbc94645a7884792496/roles/96eb47d9c9e341ab91ecffdb8ea813c8 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:34:12.591583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdcdeae-5478-4f28-87b2-2d328057b5bc None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:12.592622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdcdeae-5478-4f28-87b2-2d328057b5bc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:12.593114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdcdeae-5478-4f28-87b2-2d328057b5bc None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:12.595845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 458/945] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:12.601928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5848cde-4dd5-4126-8d36-63f5602e37d1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:12.602892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5848cde-4dd5-4126-8d36-63f5602e37d1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:12.603371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5848cde-4dd5-4126-8d36-63f5602e37d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:12.623697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a5848cde-4dd5-4126-8d36-63f5602e37d1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:12.625477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a5848cde-4dd5-4126-8d36-63f5602e37d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:12.699095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 488/946] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:12.967702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdcdeae-5478-4f28-87b2-2d328057b5bc None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:13.071715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdcdeae-5478-4f28-87b2-2d328057b5bc None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:13.106462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:13.107061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:13.107337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:13.107606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:13.108249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:13.110029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:13.112679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING keystone.server.flask.common [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 04 08:34:13.127790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:13.191121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.sql.core [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 08:34:13.191121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 04 08:34:13.191121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [parameters: {'id': '6b814e4803f147bea048ccbb46c98e94', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 04 08:34:13.191121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=88204) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6b814e4803f147bea048ccbb46c98e94.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6b814e4803f147bea048ccbb46c98e94. Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:34:13.209822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result.read() Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 04 08:34:13.210715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application yield resource Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise self.value Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application session.commit() Oct 04 08:34:13.211626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.session.flush() Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application compat.raise_( Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 08:34:13.213172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 08:34:13.215427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result.read() Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:34:13.217184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [parameters: {'id': '6b814e4803f147bea048ccbb46c98e94', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:13.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:13.224303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6b814e4803f147bea048ccbb46c98e94. Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:13.226546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 459/947] 10.222.0.27 () {60 vars in 1140 bytes} [Tue Oct 4 08:34:12 2022] POST /identity/v3/projects => generated 262 bytes in 244 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:13.227818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5848cde-4dd5-4126-8d36-63f5602e37d1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:13.247980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:13.337563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5848cde-4dd5-4126-8d36-63f5602e37d1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:13.377749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6959d94-267d-4dc3-bff6-fbc8f3d16ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:13.379541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6959d94-267d-4dc3-bff6-fbc8f3d16ea1 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:13.379541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6959d94-267d-4dc3-bff6-fbc8f3d16ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:13.379541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6959d94-267d-4dc3-bff6-fbc8f3d16ea1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:13.386388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6959d94-267d-4dc3-bff6-fbc8f3d16ea1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:13.388051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6959d94-267d-4dc3-bff6-fbc8f3d16ea1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-f6959d94-267d-4dc3-bff6-fbc8f3d16ea1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:13.402576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:13.403438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:13.404290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:13.410673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 489/948] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:13.412887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:13.522627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf5d181-3161-412c-85ae-b9c187ff3d77 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:13.558819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368df69d-6c4e-46c7-85dc-e99181174209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:13.559787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368df69d-6c4e-46c7-85dc-e99181174209 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:13.560259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368df69d-6c4e-46c7-85dc-e99181174209 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:13.560743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368df69d-6c4e-46c7-85dc-e99181174209 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:13.561588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368df69d-6c4e-46c7-85dc-e99181174209 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:13.562077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368df69d-6c4e-46c7-85dc-e99181174209 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:13.564127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368df69d-6c4e-46c7-85dc-e99181174209 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:13.585090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 460/949] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:34:13 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:14.113949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db57cce6-095e-4bef-8bbe-bef7c320ad6f None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.114647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db57cce6-095e-4bef-8bbe-bef7c320ad6f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.114647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db57cce6-095e-4bef-8bbe-bef7c320ad6f None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.116115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 490/950] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:14.122154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f2c67-9ce5-489b-ad5a-8ee3bb29ffb6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.122623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f2c67-9ce5-489b-ad5a-8ee3bb29ffb6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.122728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810f2c67-9ce5-489b-ad5a-8ee3bb29ffb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.142703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-810f2c67-9ce5-489b-ad5a-8ee3bb29ffb6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:14.142889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae1ed1f-3dff-4aaa-a064-d03bfe57f28b None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.143113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae1ed1f-3dff-4aaa-a064-d03bfe57f28b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.143371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae1ed1f-3dff-4aaa-a064-d03bfe57f28b None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.144228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-810f2c67-9ce5-489b-ad5a-8ee3bb29ffb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:14.144963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 491/951] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:14.150201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b3ed9d-62ea-4129-a520-e43355c6ff28 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.150502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b3ed9d-62ea-4129-a520-e43355c6ff28 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.150778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b3ed9d-62ea-4129-a520-e43355c6ff28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.171829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-30b3ed9d-62ea-4129-a520-e43355c6ff28 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:14.173140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-30b3ed9d-62ea-4129-a520-e43355c6ff28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:14.213611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:14.229927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 461/952] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:14.266661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 492/953] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:14.329781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186561e-9cdf-4d00-9346-704ebf602e4c None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.330178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186561e-9cdf-4d00-9346-704ebf602e4c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.330481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186561e-9cdf-4d00-9346-704ebf602e4c None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.332161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 462/954] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:14.338138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6066e89-49f5-400c-9952-ea9b57d55748 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.338578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6066e89-49f5-400c-9952-ea9b57d55748 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.338946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6066e89-49f5-400c-9952-ea9b57d55748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.347107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:14.358384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c6066e89-49f5-400c-9952-ea9b57d55748 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:14.359836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c6066e89-49f5-400c-9952-ea9b57d55748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:14.434549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 493/955] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:14.520372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8186561e-9cdf-4d00-9346-704ebf602e4c None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:14.568216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6066e89-49f5-400c-9952-ea9b57d55748 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:14.640368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8186561e-9cdf-4d00-9346-704ebf602e4c None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:14.679748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:14.680116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.680395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.680661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] PATH_INFO: `/v3/services/compute` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.681287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:14.682597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:14.691862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6066e89-49f5-400c-9952-ea9b57d55748 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:14.692771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:14.693804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:14.694697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:14.698947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 463/956] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:14 2022] GET /identity/v3/services/compute => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:14.699778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:14.741952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:14.742854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.743271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.743980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.744701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:14.746776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:14.749331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:14.757741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:14.764672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:14.765739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 494/957] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:14 2022] GET /identity/v3/services/object-store => generated 93 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:14.767039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:14.767108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:14.811721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348afe3-54d5-4269-ab40-158c2209b143 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:14.853108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:14.853762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.854036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.854300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.854952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] RBAC: Adding query filter params (name=compute) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:14.855341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:14.856828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:14.862923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 464/958] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:34:14 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:14.866702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:14.869104 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11a3553-ad8c-4819-bc74-faf92c169272 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:14.906259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:14.907426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:14.907711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:14.907986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:14.918676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:14.925060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:14.926011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:14.926873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 495/959] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Oct 4 08:34:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:14.928038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:14.987186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653ed96-5d51-426c-83dc-5564d1dca0b7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:14.999667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:15.012194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:15.030798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.032466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.032755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.033018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.033645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:15.033945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.035320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.040446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 465/960] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:14 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:15.043774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cd6366-6e65-45d5-853c-018eabf773f7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:15.045854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:15.081136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.081861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.082140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.082418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.083069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] RBAC: Adding query filter params (type=compute) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:15.083374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.084738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.090024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 496/961] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:34:14 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:15.090832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:15.155077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0152f7e-f94b-4e6a-ac6e-544f4b450a08 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:15.197774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.198907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.199396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.200343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.201581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:15.202246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.204056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.214229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 466/962] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Oct 4 08:34:15 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:15.221593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:15.232464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfc6a81-8466-4654-9812-30eb6ff7d248 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:15.263076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:15.278323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.279226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.279685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.280096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.280884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:15.281863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.283625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.288516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:15.290511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 497/963] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:15 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:15.297660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:15.351784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d5c77e-cfc2-42ac-9e10-1df26feefabc None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:15.403033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.403985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.404490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.405525 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.419482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.419926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fc481e-e635-4696-b92e-1a30db7522e4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:15.423572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:15.441979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:15.443240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:15.444128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:15.444128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:15.444128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:15.446625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 467/964] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:15.449056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:15.463117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.464054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.464718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.465220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.466114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.467725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.558077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:15.565764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5083858-a8aa-41b8-aa9f-289558ff8597 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:15.566352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 498/965] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:34:15 2022] POST /identity/v3/endpoints => generated 349 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:15.613965 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.614614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.614890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.615160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.615810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:15.616109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.617553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.635348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 468/966] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:15.641354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:15.699705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:15.775040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c2461-2658-4167-b088-ef51d125f9ca None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:15.818774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a052ac6-3b7d-46a6-9183-18f995814670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:15.819339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a052ac6-3b7d-46a6-9183-18f995814670 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.819612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a052ac6-3b7d-46a6-9183-18f995814670 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.819879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a052ac6-3b7d-46a6-9183-18f995814670 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.828912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a052ac6-3b7d-46a6-9183-18f995814670 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:15.830358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a052ac6-3b7d-46a6-9183-18f995814670 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-3a052ac6-3b7d-46a6-9183-18f995814670 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:34:15.842308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:15.843302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:15.846092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 499/967] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:34:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:15.849988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:15.897600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d221610-6b51-4f7a-a0f5-de892597951e None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.897870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d221610-6b51-4f7a-a0f5-de892597951e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.898141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d221610-6b51-4f7a-a0f5-de892597951e None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.899753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 500/968] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:15.904996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd23ea1-e557-48a1-a801-47e359db7ee8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:15.905309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd23ea1-e557-48a1-a801-47e359db7ee8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:15.905544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd23ea1-e557-48a1-a801-47e359db7ee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:15.925831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8bd23ea1-e557-48a1-a801-47e359db7ee8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:15.927186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8bd23ea1-e557-48a1-a801-47e359db7ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:15.973446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c16fe-bd85-4351-95e4-11fb06186bee None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.006696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 501/969] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:16.016862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.017722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.018201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.018720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.019642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:16.020120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.021763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:16.041641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 469/970] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Oct 4 08:34:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:16.047789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd23ea1-e557-48a1-a801-47e359db7ee8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:16.089760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:16.166772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd23ea1-e557-48a1-a801-47e359db7ee8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.207576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.208188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.208467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.208741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.283170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:16.284920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.286380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:16.297186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:16.318244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:16.406700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 502/971] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:16 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:16.414442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f1b5bd-baa6-4056-b382-b48c6671ea05 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.414619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:16.455446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.456711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.457146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.457896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.459020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.460592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:16.506837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.sql.core [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 08:34:16.506837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 04 08:34:16.506837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [parameters: {'user_id': '05ce1dca046646aca39dcbea54e82518', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 04 08:34:16.506837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=88204) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result.read() Oct 04 08:34:16.517285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:34:16.518176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application yield resource Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise self.value Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application session.commit() Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 08:34:16.519449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.session.flush() Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application compat.raise_( Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 08:34:16.520319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:34:16.521236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application result.read() Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:34:16.522074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [parameters: {'user_id': '05ce1dca046646aca39dcbea54e82518', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:16.522946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 04 08:34:16.523716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 04 08:34:16.527620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 04 08:34:16.527620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 04 08:34:16.527620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 08:34:16.527620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 08:34:16.527620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 04 08:34:16.527620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:16.528187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1148b917-829e-4d33-ab1b-2e47c4826ef8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.528298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 470/972] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Oct 4 08:34:16 2022] POST /identity/v3/users => generated 160 bytes in 242 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:16.530450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:16.557095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:16.563661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.564323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.564704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.565007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.576348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.578829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:16.594813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:16.595787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:16.596657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:16.596657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:16.596657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:16.596657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 503/973] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:16.601599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:16.613832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:16.645148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ad4f0f-bf68-4605-8f44-974b95564b15 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.683657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.684344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.684720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.685108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.693060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.698463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:34:16.714218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:16.715385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:34:16.716616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:16.723254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 471/974] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:34:16 2022] GET /identity/v3/users/cinder => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:16.724376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:16.738943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bccaddc-f445-435f-9e2b-b5f51f49049b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.784035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.785147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.785973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.786390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.787429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:16.788102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.790330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:16.806603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 504/975] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:16.811874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:16.842431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc711b4-a360-4674-ba60-8b466d8c4f10 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.869159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:16.874029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:16.875883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.876763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.877064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.877435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.878246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:16.878707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.880287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:16.926743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 472/976] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Oct 4 08:34:16 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:16.936557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51225fab-8f98-4ada-9a9a-a8f2b22435b8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:16.981865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:16.982844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:16.983276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:16.983696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:16.993308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:16.995397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:17.009729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:17.010885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:17.011992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:17.014147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 505/977] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:17.020925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:17.141552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80de2557-12d0-43b2-9bff-c0821f7b9cf2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:17.179030 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6976d4-6a7c-4075-951b-28c44c4d216e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:17.179647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6976d4-6a7c-4075-951b-28c44c4d216e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:17.179924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6976d4-6a7c-4075-951b-28c44c4d216e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:17.180189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6976d4-6a7c-4075-951b-28c44c4d216e None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:17.180880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6976d4-6a7c-4075-951b-28c44c4d216e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:17.181184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6976d4-6a7c-4075-951b-28c44c4d216e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:17.182619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6976d4-6a7c-4075-951b-28c44c4d216e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:17.195985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 473/978] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:17.201931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:17.320189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b53c6a7-1307-4e19-a263-23d559e399e0 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:17.322981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:17.361260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d67735-88d7-4b3f-b163-8e0390911b33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:17.361870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d67735-88d7-4b3f-b163-8e0390911b33 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:17.362148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d67735-88d7-4b3f-b163-8e0390911b33 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:17.362438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d67735-88d7-4b3f-b163-8e0390911b33 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:17.363168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d67735-88d7-4b3f-b163-8e0390911b33 None admin] RBAC: Adding query filter params (role.id=96eb47d9c9e341ab91ecffdb8ea813c8, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=c9cff7d749314dbc94645a7884792496) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:17.363471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d67735-88d7-4b3f-b163-8e0390911b33 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:17.364910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d67735-88d7-4b3f-b163-8e0390911b33 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:17.376054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 506/979] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:17 2022] GET /identity/v3/role_assignments?user.id=c9cff7d749314dbc94645a7884792496&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=96eb47d9c9e341ab91ecffdb8ea813c8 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:17.450211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac95a96-e479-44e8-ac26-4892ee2237ff None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:17.450743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac95a96-e479-44e8-ac26-4892ee2237ff None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:17.450743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac95a96-e479-44e8-ac26-4892ee2237ff None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:17.452070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 474/980] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:17.458896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb83ba50-4250-4c8e-85e8-1ef4b760f7a1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:17.459192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb83ba50-4250-4c8e-85e8-1ef4b760f7a1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:17.459453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb83ba50-4250-4c8e-85e8-1ef4b760f7a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:17.479978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-cb83ba50-4250-4c8e-85e8-1ef4b760f7a1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:17.481201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cb83ba50-4250-4c8e-85e8-1ef4b760f7a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:17.562539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 507/981] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:17.839490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac95a96-e479-44e8-ac26-4892ee2237ff None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:17.892614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:17.912248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4e7429-a5a7-428e-947d-18c3c48c1f3a None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:17.912621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4e7429-a5a7-428e-947d-18c3c48c1f3a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:17.913063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4e7429-a5a7-428e-947d-18c3c48c1f3a None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:17.914377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 508/982] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:17.919598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b8f1b-57c8-49c2-b482-94eac8584bf7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:17.919875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b8f1b-57c8-49c2-b482-94eac8584bf7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:17.920145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b8f1b-57c8-49c2-b482-94eac8584bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:17.938311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-bb1b8f1b-57c8-49c2-b482-94eac8584bf7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:17.939656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-bb1b8f1b-57c8-49c2-b482-94eac8584bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:17.948158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac95a96-e479-44e8-ac26-4892ee2237ff None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:17.985645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:17.986766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:17.987148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:17.987425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:17.988076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:17.989639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:18.009720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 509/983] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:18.077859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 475/984] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:34:17 2022] POST /identity/v3/services => generated 242 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:18.280646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1b8f1b-57c8-49c2-b482-94eac8584bf7 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:18.311107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:18.391536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1b8f1b-57c8-49c2-b482-94eac8584bf7 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:18.431771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:18.432479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:18.432818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:18.433147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:18.433781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:18.435406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:18.444831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:18.446030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:18.447252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:18.450472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 510/985] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:18 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:18.453387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:18.554848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4dcf34-a87f-421d-99fb-3271dd6f6d46 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:18.565862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:18.595610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:18.596228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:18.596541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:18.596783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:18.597425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:18.598136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:18.600217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:18.605370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 476/986] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:18 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:18.611902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:18.702379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:18.714302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511b11f0-4156-4d23-9c99-6ede2585d3b8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:18.747237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:18.748177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:18.748468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:18.748737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:18.749356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:18.749669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:18.751945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:18.761749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 511/987] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:18 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:18.763385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:18.866040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d27ccc8-6f12-45ce-a2ca-73d366be9df2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:18.898740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84747a0c-36cc-44e1-8ba6-c48d84feab7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:18.899346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84747a0c-36cc-44e1-8ba6-c48d84feab7c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:18.899622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84747a0c-36cc-44e1-8ba6-c48d84feab7c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:18.899890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84747a0c-36cc-44e1-8ba6-c48d84feab7c None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:18.900592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84747a0c-36cc-44e1-8ba6-c48d84feab7c None admin] RBAC: Adding query filter params (interface=admin, service_id=a89fd48941f5433a85abac6f65759ee7, region_id=RegionOne) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:18.900899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84747a0c-36cc-44e1-8ba6-c48d84feab7c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:18.902244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84747a0c-36cc-44e1-8ba6-c48d84feab7c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:18.912810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 477/988] 10.222.0.27 () {56 vars in 1249 bytes} [Tue Oct 4 08:34:18 2022] GET /identity/v3/endpoints?service_id=a89fd48941f5433a85abac6f65759ee7&interface=admin®ion_id=RegionOne => generated 193 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:18.913972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:19.016417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae68cd-d210-4373-9355-61770fdf6b9d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:19.050271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa58286-55f2-481c-87a1-334ad6089e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:19.050903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa58286-55f2-481c-87a1-334ad6089e18 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:19.051184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa58286-55f2-481c-87a1-334ad6089e18 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:19.051451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa58286-55f2-481c-87a1-334ad6089e18 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:19.052048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa58286-55f2-481c-87a1-334ad6089e18 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:19.053408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa58286-55f2-481c-87a1-334ad6089e18 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:19.064682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 512/989] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:34:18 2022] GET /identity/v3/services => generated 1727 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:34:19.595200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e010e9c-af29-471d-9fad-2b20d5f23db7 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:19.595711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e010e9c-af29-471d-9fad-2b20d5f23db7 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:19.603942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e010e9c-af29-471d-9fad-2b20d5f23db7 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:19.603942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 478/990] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:19.604128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b00d924-3714-4d4f-83f4-0814b753eb52 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:19.604128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b00d924-3714-4d4f-83f4-0814b753eb52 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:19.604128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b00d924-3714-4d4f-83f4-0814b753eb52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:19.621790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1b00d924-3714-4d4f-83f4-0814b753eb52 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:19.623001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1b00d924-3714-4d4f-83f4-0814b753eb52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:19.634598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:19.694291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 513/991] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:19 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:19.969401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e010e9c-af29-471d-9fad-2b20d5f23db7 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:20.011269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:20.071373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e010e9c-af29-471d-9fad-2b20d5f23db7 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:20.104917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:20.105543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:20.105820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:20.106091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] PATH_INFO: `/v3/services/image` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:20.106695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:20.108049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:20.116996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:20.117955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:20.119455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:20.122723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 479/992] 10.222.0.27 () {56 vars in 1105 bytes} [Tue Oct 4 08:34:19 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:20.124216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b00d924-3714-4d4f-83f4-0814b753eb52 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:20.231046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b00d924-3714-4d4f-83f4-0814b753eb52 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:20.265020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df47af7-eddb-4d9f-97d3-de4385fc3232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:20.266623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df47af7-eddb-4d9f-97d3-de4385fc3232 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:20.266903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df47af7-eddb-4d9f-97d3-de4385fc3232 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:20.267164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df47af7-eddb-4d9f-97d3-de4385fc3232 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:20.267760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df47af7-eddb-4d9f-97d3-de4385fc3232 None admin] RBAC: Adding query filter params (name=image) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:20.268058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df47af7-eddb-4d9f-97d3-de4385fc3232 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:20.269533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df47af7-eddb-4d9f-97d3-de4385fc3232 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:20.279078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 514/993] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:20 2022] GET /identity/v3/services?name=image => generated 122 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:20.280148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:20.296518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d099b1-bc2d-4e60-8034-619fb12627b6 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:20.296797 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d099b1-bc2d-4e60-8034-619fb12627b6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:20.297195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d099b1-bc2d-4e60-8034-619fb12627b6 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:20.298549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 515/994] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:20.303442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e4e2ba-e558-4704-9b0c-8a2678c69e56 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:20.303724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e4e2ba-e558-4704-9b0c-8a2678c69e56 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:20.303999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e4e2ba-e558-4704-9b0c-8a2678c69e56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:20.322830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-62e4e2ba-e558-4704-9b0c-8a2678c69e56 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:20.324024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-62e4e2ba-e558-4704-9b0c-8a2678c69e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:20.327720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:20.403476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7760ecf6-57a6-4081-96db-77383a92792b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:20.404062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 516/995] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:20 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:20.405829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:20.450867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:20.451826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:20.452140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:20.452408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:20.453030 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] RBAC: Adding query filter params (type=image) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:20.453330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:20.454720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:20.466673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 480/996] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:20 2022] GET /identity/v3/services?type=image => generated 122 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:20.679205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e4e2ba-e558-4704-9b0c-8a2678c69e56 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:20.783507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e4e2ba-e558-4704-9b0c-8a2678c69e56 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:20.818472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:20.819107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:20.819381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:20.819649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] PATH_INFO: `/v3/services/compute` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:20.820225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:20.821663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:20.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:20.831759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:20.833020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:20.834492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 517/997] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:20.838553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:20.942911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb6372e-d479-4265-9761-0a9a9912eac6 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:20.975842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:20.976445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:20.976715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:20.976978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:20.977596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] RBAC: Adding query filter params (name=compute) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:20.977891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:20.979255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:20.984927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 481/998] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:34:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:20.990530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:20.994639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:21.092187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c15d45b-5e68-49bb-b36e-fd988a3b25f7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:21.128298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96f3062-3408-47ba-919f-373283422fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:21.128919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96f3062-3408-47ba-919f-373283422fee None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:21.129193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96f3062-3408-47ba-919f-373283422fee None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:21.129457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96f3062-3408-47ba-919f-373283422fee None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:21.130098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96f3062-3408-47ba-919f-373283422fee None admin] RBAC: Adding query filter params (type=compute) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:21.130394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96f3062-3408-47ba-919f-373283422fee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:21.131981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96f3062-3408-47ba-919f-373283422fee None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:21.137939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 518/999] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:34:20 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:21.144137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:21.177978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:21.249147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e803b07d-b545-4dde-be90-c8f2eade4622 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:21.261756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ccdea2-1b89-42e4-90a2-b1476b353c69 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:21.262039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ccdea2-1b89-42e4-90a2-b1476b353c69 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:21.262311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ccdea2-1b89-42e4-90a2-b1476b353c69 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:21.263891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 519/1000] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:21.268843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612687c-5db6-46ed-a333-5cf1d324b466 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:21.269097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612687c-5db6-46ed-a333-5cf1d324b466 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:21.269382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612687c-5db6-46ed-a333-5cf1d324b466 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:21.287167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:21.287537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a612687c-5db6-46ed-a333-5cf1d324b466 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:21.287983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:21.288357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:21.288721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:21.288817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a612687c-5db6-46ed-a333-5cf1d324b466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:21.289595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] RBAC: Adding query filter params (interface=public, service_id=b615c4541a0f40839a4880e1bc1c5892, region_id=RegionOne) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:21.289995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:21.291462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:21.296392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 482/1001] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:34:21 2022] GET /identity/v3/endpoints?service_id=b615c4541a0f40839a4880e1bc1c5892&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:21.302518 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:21.361350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:21.367808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 520/1002] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:21 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:21.418274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:21.425531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbe5682-e9eb-486e-a5f4-5b9675ebd7f7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:21.460836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:21.461494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:21.461773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:21.462046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:21.462723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:21.464087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:21.471777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 483/1003] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:34:21 2022] GET /identity/v3/services => generated 1727 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:34:21.646003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a612687c-5db6-46ed-a333-5cf1d324b466 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:21.752940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a612687c-5db6-46ed-a333-5cf1d324b466 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:21.787398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:21.788025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:21.788300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:21.788567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:21.789140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:21.790517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:21.799609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:21.800600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:21.801441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 521/1004] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Oct 4 08:34:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:21.807205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:21.908690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fb0a10-a2c7-4498-a911-1b5a51513ea6 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:21.942196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:21.942873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:21.943153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:21.943421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:21.944035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:21.944329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:21.945677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:21.955775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 484/1005] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:21.957192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:22.021603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:22.060668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd1bd9f-5c3e-4df2-abef-ff077c8b2b0b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:22.095266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6e9d3e-d8bb-4850-a54d-b2d0a71faf3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:22.095871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6e9d3e-d8bb-4850-a54d-b2d0a71faf3e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:22.096149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6e9d3e-d8bb-4850-a54d-b2d0a71faf3e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:22.096418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6e9d3e-d8bb-4850-a54d-b2d0a71faf3e None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:22.097100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6e9d3e-d8bb-4850-a54d-b2d0a71faf3e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:22.097402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6e9d3e-d8bb-4850-a54d-b2d0a71faf3e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:22.098850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6e9d3e-d8bb-4850-a54d-b2d0a71faf3e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:22.108236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 522/1006] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Oct 4 08:34:21 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:22.109816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:22.135218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:22.211565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674973db-75ca-4137-8c37-1ca7dd1dfca6 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:22.246058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd343d7-da4a-4ee1-bb34-f8c1573226a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:22.246808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd343d7-da4a-4ee1-bb34-f8c1573226a5 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:22.247153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd343d7-da4a-4ee1-bb34-f8c1573226a5 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:22.247486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd343d7-da4a-4ee1-bb34-f8c1573226a5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:22.248211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd343d7-da4a-4ee1-bb34-f8c1573226a5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:22.250195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd343d7-da4a-4ee1-bb34-f8c1573226a5 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:22.330937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 485/1007] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:34:22 2022] POST /identity/v3/endpoints => generated 325 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:34:22.679743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dc7a7e-393c-471f-9580-4a68e1755f80 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:22.680026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dc7a7e-393c-471f-9580-4a68e1755f80 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:22.680297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dc7a7e-393c-471f-9580-4a68e1755f80 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:22.681794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 523/1008] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:22.686847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de3c76d-4446-4b2c-b6c0-841e661e6f52 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:22.687668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de3c76d-4446-4b2c-b6c0-841e661e6f52 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:22.688494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de3c76d-4446-4b2c-b6c0-841e661e6f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:22.707529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6de3c76d-4446-4b2c-b6c0-841e661e6f52 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:22.709134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6de3c76d-4446-4b2c-b6c0-841e661e6f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:22.783215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 486/1009] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:22 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:23.059567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dc7a7e-393c-471f-9580-4a68e1755f80 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:23.185091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dc7a7e-393c-471f-9580-4a68e1755f80 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:23.217677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:23.227585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a100aff-dbfc-493f-95f1-5397783f6dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:23.228468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a100aff-dbfc-493f-95f1-5397783f6dc4 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:23.228917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a100aff-dbfc-493f-95f1-5397783f6dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:23.229341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a100aff-dbfc-493f-95f1-5397783f6dc4 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:23.231536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a100aff-dbfc-493f-95f1-5397783f6dc4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:23.233569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a100aff-dbfc-493f-95f1-5397783f6dc4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:23.336139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 524/1010] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:34:23 2022] POST /identity/v3/services => generated 242 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:34:23.792928 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dbddf5-6a64-4fa5-b899-229e80db3ae1 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:23.793643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dbddf5-6a64-4fa5-b899-229e80db3ae1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:23.793643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dbddf5-6a64-4fa5-b899-229e80db3ae1 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:23.794904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 487/1011] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:23.800914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e5a27-42cb-425f-9292-e68719ee3072 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:23.801192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e5a27-42cb-425f-9292-e68719ee3072 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:23.801470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e5a27-42cb-425f-9292-e68719ee3072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:23.824317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8b7e5a27-42cb-425f-9292-e68719ee3072 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:23.831570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8b7e5a27-42cb-425f-9292-e68719ee3072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:23.931250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 525/1012] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:23 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:24.213364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dbddf5-6a64-4fa5-b899-229e80db3ae1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:24.224643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:24.318315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dbddf5-6a64-4fa5-b899-229e80db3ae1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:24.354216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:24.354872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:24.355245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:24.355514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] PATH_INFO: `/v3/services/compute` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:24.356090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:24.357495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:24.366390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:24.367367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:34:24.368190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:24.370563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 488/1013] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:24 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:24.377767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7e5a27-42cb-425f-9292-e68719ee3072 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:24.479240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7e5a27-42cb-425f-9292-e68719ee3072 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:24.492505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:24.514162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:24.514803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:24.515081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:24.515352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:24.515981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] RBAC: Adding query filter params (name=compute) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:24.516284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:24.517707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:24.527014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 526/1014] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:34:24 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:24.528210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:24.605901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:24.635579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08aa885-cbf4-419f-8d84-724fec5e1435 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:24.671216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4232fb-01b5-49b8-be19-cf7c90897cbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:24.672369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4232fb-01b5-49b8-be19-cf7c90897cbb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:24.672655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4232fb-01b5-49b8-be19-cf7c90897cbb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:24.672896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4232fb-01b5-49b8-be19-cf7c90897cbb None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:24.673544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4232fb-01b5-49b8-be19-cf7c90897cbb None admin] RBAC: Adding query filter params (type=compute) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:24.673844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4232fb-01b5-49b8-be19-cf7c90897cbb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:24.675281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4232fb-01b5-49b8-be19-cf7c90897cbb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:24.681228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 489/1015] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Oct 4 08:34:24 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:24.686924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:24.788241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d5a492-b738-4121-8b18-deef54eb6faa None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:24.788572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d5a492-b738-4121-8b18-deef54eb6faa None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:24.788780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d5a492-b738-4121-8b18-deef54eb6faa None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:24.789869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58942c9c-0a9c-4645-a70c-7fe03d2e0e36 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:24.793872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 490/1016] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:24.795654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b61f42-b1c2-4735-8b47-3ba5def2c08a None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:24.795926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b61f42-b1c2-4735-8b47-3ba5def2c08a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:24.796195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b61f42-b1c2-4735-8b47-3ba5def2c08a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:24.815101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-13b61f42-b1c2-4735-8b47-3ba5def2c08a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:24.816495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-13b61f42-b1c2-4735-8b47-3ba5def2c08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:24.834906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:24.835789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:24.836267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:24.836549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:24.837236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:24.838600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:24.892020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 491/1017] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:24 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:24.922435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 527/1018] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:34:24 2022] POST /identity/v3/endpoints => generated 334 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:25.167271 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b61f42-b1c2-4735-8b47-3ba5def2c08a None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:25.237607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:25.272060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b61f42-b1c2-4735-8b47-3ba5def2c08a None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:25.313639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9db90-e771-4742-abd1-1df32a2d920b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:25.314266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9db90-e771-4742-abd1-1df32a2d920b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:25.314564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9db90-e771-4742-abd1-1df32a2d920b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:25.314828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9db90-e771-4742-abd1-1df32a2d920b None admin] PATH_INFO: `/v3/domains/default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:25.315921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9db90-e771-4742-abd1-1df32a2d920b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:25.317328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9db90-e771-4742-abd1-1df32a2d920b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:25.319099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 492/1019] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:25.324602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:25.356024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:25.438321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd393f80-4173-4764-a5c7-a0b7e7f8ec10 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:25.473287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1769f0f6-ca82-407d-bb83-d55efaa3f2c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:25.473904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1769f0f6-ca82-407d-bb83-d55efaa3f2c8 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:25.474183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1769f0f6-ca82-407d-bb83-d55efaa3f2c8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:25.474476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1769f0f6-ca82-407d-bb83-d55efaa3f2c8 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:25.475132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1769f0f6-ca82-407d-bb83-d55efaa3f2c8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:25.476563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1769f0f6-ca82-407d-bb83-d55efaa3f2c8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:25.586921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 528/1020] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:34:25 2022] POST /identity/v3/projects => generated 311 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:25.651431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8758361e-71f2-44c5-a67e-86edc66f9cae None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:25.651752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8758361e-71f2-44c5-a67e-86edc66f9cae None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:25.651984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8758361e-71f2-44c5-a67e-86edc66f9cae None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:25.653492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 493/1021] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:25.660162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85358c8-34c7-4e22-aabc-5d663d20e43c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:25.660839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85358c8-34c7-4e22-aabc-5d663d20e43c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:25.661309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85358c8-34c7-4e22-aabc-5d663d20e43c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:25.681603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e85358c8-34c7-4e22-aabc-5d663d20e43c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:25.683289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e85358c8-34c7-4e22-aabc-5d663d20e43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:25.762455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 529/1022] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:25 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:26.039897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8758361e-71f2-44c5-a67e-86edc66f9cae None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:26.148247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8758361e-71f2-44c5-a67e-86edc66f9cae None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:26.177558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:26.183644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:26.184564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:26.185054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:26.185505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] PATH_INFO: `/v3/services/image` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:26.186270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:26.187855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:26.196891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:26.198017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:26.198967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:26.200222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 494/1023] 10.222.0.27 () {56 vars in 1105 bytes} [Tue Oct 4 08:34:26 2022] GET /identity/v3/services/image => generated 86 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:26.206875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85358c8-34c7-4e22-aabc-5d663d20e43c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:26.223570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:26.310791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85358c8-34c7-4e22-aabc-5d663d20e43c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:26.347015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:26.347623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:26.347893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:26.348158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:26.348785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] RBAC: Adding query filter params (name=image) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:26.349082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:26.350453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:26.360152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 530/1024] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:26 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:26.361220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:26.467497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3f5697-9497-43e0-aec4-5a53b54b1a8e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:26.500425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:26.501032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:26.501303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:26.501567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:26.502185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] RBAC: Adding query filter params (type=image) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:26.502507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:26.503858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:26.513308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 495/1025] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:26 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:26.514889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:26.597277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:26.618778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de83ac7-ad26-46a3-9d8c-9b35ac623523 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:26.652454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64971cf0-28a0-48e3-959f-45629a4231dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:26.659996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64971cf0-28a0-48e3-959f-45629a4231dd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:26.659996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64971cf0-28a0-48e3-959f-45629a4231dd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:26.659996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64971cf0-28a0-48e3-959f-45629a4231dd None admin] PATH_INFO: `/v3/endpoints` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:26.659996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64971cf0-28a0-48e3-959f-45629a4231dd None admin] RBAC: Adding query filter params (interface=public, service_id=f6630dcef39c4cf68bc650aa00799eeb, region_id=RegionOne) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:26.659996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64971cf0-28a0-48e3-959f-45629a4231dd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:26.659996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64971cf0-28a0-48e3-959f-45629a4231dd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:26.661092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 531/1026] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:34:26 2022] GET /identity/v3/endpoints?service_id=f6630dcef39c4cf68bc650aa00799eeb&interface=public®ion_id=RegionOne => generated 194 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:26.666952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:26.727522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:26.770118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56e2c3-e62c-43f6-bad6-ad174552f683 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:26.803529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e074f8c-99ed-45fd-82f3-44f618a3ce29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:26.804131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e074f8c-99ed-45fd-82f3-44f618a3ce29 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:26.804404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e074f8c-99ed-45fd-82f3-44f618a3ce29 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:26.804669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e074f8c-99ed-45fd-82f3-44f618a3ce29 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:26.805265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e074f8c-99ed-45fd-82f3-44f618a3ce29 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:26.806636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e074f8c-99ed-45fd-82f3-44f618a3ce29 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:26.814051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 496/1027] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:34:26 2022] GET /identity/v3/services => generated 1957 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:34:27.305821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77e40b-b0b7-4af6-a843-e111030437fb None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:27.306107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77e40b-b0b7-4af6-a843-e111030437fb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:27.306614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77e40b-b0b7-4af6-a843-e111030437fb None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:27.308127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 532/1028] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:27.314023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c966fe6-24cf-4fca-9281-d8e9540a69ca None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:27.314517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c966fe6-24cf-4fca-9281-d8e9540a69ca None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:27.314605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c966fe6-24cf-4fca-9281-d8e9540a69ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:27.333209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5c966fe6-24cf-4fca-9281-d8e9540a69ca None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:27.334422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5c966fe6-24cf-4fca-9281-d8e9540a69ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:27.413199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 497/1029] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:27 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:27.720839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b77e40b-b0b7-4af6-a843-e111030437fb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:27.832446 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b77e40b-b0b7-4af6-a843-e111030437fb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:27.860535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbff50d6-949e-41a5-9e4b-5ab1ff68a81a None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:27.860813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbff50d6-949e-41a5-9e4b-5ab1ff68a81a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:27.861088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbff50d6-949e-41a5-9e4b-5ab1ff68a81a None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:27.862441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 498/1030] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:27.867281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14930a81-3375-479e-9294-33953d506c97 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:27.867564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14930a81-3375-479e-9294-33953d506c97 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:27.867836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14930a81-3375-479e-9294-33953d506c97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:27.868385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:27.869279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:27.869555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:27.869823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:27.883392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:27.886592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-14930a81-3375-479e-9294-33953d506c97 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:27.887805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-14930a81-3375-479e-9294-33953d506c97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:27.891494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:27.894293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:27.895371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 533/1031] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Oct 4 08:34:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:27.900998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:27.904390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:27.952162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:27.973474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 499/1032] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:27 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:28.013876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3320fc16-0bcd-47b7-b815-6376914fa0cd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.049107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.049715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.049994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.050265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:28.050952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:28.051252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:28.052614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:28.060755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 534/1033] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Oct 4 08:34:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:28.066464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14930a81-3375-479e-9294-33953d506c97 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:28.165789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14930a81-3375-479e-9294-33953d506c97 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.199624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.200714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.201210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.201687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:28.216215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:28.217693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:28.236527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:28.237537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:28.238876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:28.238876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:28.238876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:28.242373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:28.242547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 500/1034] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:28.243057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:28.285837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:28.313631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:28.353442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0bb570-7a25-4cdf-8d44-ac628c8b74c6 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.354068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8a7e25-b139-4d56-994e-50e67ad857bd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.391012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.392009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.392501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.392987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:28.393837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:28.394308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:28.396028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:28.396249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.396892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.397172 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.397442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:28.398642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:28.400006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:28.401726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 535/1035] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:28.407166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:28.413772 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 501/1036] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:28.414608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:28.514959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67ec8d8-a7f0-4386-abb0-a16a8f45d2dc None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.515990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:28.522972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b619c45e-5901-419d-80aa-c4ced6107759 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.549267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:28.553946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.554590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.554871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.555144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:28.555788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:28.557219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:28.563929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.564554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.564831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.565093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] PATH_INFO: `/v3/users/nova` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:28.573076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:28.574549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:34:28.587305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:28.588527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:28.589475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 502/1037] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:34:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:28.594113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:28.665923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 536/1038] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:34:28 2022] POST /identity/v3/users => generated 285 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:34:28.704833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bbdd3d-a0cb-4a09-bcf4-c96430412d4c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.742831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ad26c6-ba15-4119-a6d8-1d0834987193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.743708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ad26c6-ba15-4119-a6d8-1d0834987193 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.743987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ad26c6-ba15-4119-a6d8-1d0834987193 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.744255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ad26c6-ba15-4119-a6d8-1d0834987193 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:28.745001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ad26c6-ba15-4119-a6d8-1d0834987193 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:28.745382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ad26c6-ba15-4119-a6d8-1d0834987193 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:28.746796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ad26c6-ba15-4119-a6d8-1d0834987193 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:28.790291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 503/1039] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:34:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:28.791017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:28.904025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc34ed3-fa3f-4388-9b4b-af127f09ac6f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:28.944863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:28.945477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:28.945752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:28.946022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.004890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.021458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.021458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.025120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f386fa-7d39-4ed3-8d04-0f39ae6f0aa6 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.025404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f386fa-7d39-4ed3-8d04-0f39ae6f0aa6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.025678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f386fa-7d39-4ed3-8d04-0f39ae6f0aa6 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.027179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 504/1040] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:29.032323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f9500-10a8-4fac-a14f-02d41ac9f721 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.032572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f9500-10a8-4fac-a14f-02d41ac9f721 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.032809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f9500-10a8-4fac-a14f-02d41ac9f721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.058114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d87f9500-10a8-4fac-a14f-02d41ac9f721 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:29.058114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d87f9500-10a8-4fac-a14f-02d41ac9f721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:29.061419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.127327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 505/1041] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:29 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:29.128319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 537/1042] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:28 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:29.133717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f9500-10a8-4fac-a14f-02d41ac9f721 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.235974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f9500-10a8-4fac-a14f-02d41ac9f721 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:29.271407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:29.272062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.272341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.272610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.283457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.284893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.286752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:29.299953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:29.300943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:29.301817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:29.301817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:29.301817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:29.301817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 506/1043] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:29.306786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.394520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.403422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.416499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac33d65-1a96-4b5e-bb58-762d04a73931 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:29.454227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:29.454867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.455148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.455420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.456191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:29.456492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.457859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.473342 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 538/1044] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:29.478836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.509746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a60e8e-5234-462b-8d82-5da5cffda473 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:29.550453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:29.550453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.550453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.550453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] PATH_INFO: `/v3/services/image` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.550868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.553419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:29.567432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:29.568492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:29.569312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 507/1045] 10.222.0.27 () {56 vars in 1105 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/services/image => generated 86 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:29.575310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.603242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e69363-3e38-44dc-9fad-5b6e9400280d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:29.641286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.644003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:29.644662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.644941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.645210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.652721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.655161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:29.668022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:29.669014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:29.669909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 539/1046] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:29.675673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.686884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.690571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d265d-affb-46cc-9a2e-181a247d4ac4 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:29.727616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:29.728216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.728492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.728761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.729388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] RBAC: Adding query filter params (name=image) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:29.729690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.731096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.740569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 508/1047] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/services?name=image => generated 122 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:29.741527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.787436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d66ccb5-0280-443a-a314-d0b5f24e61fa None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:29.822109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:29.822910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.823101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.823849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.824543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:29.824847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.827178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.848958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 540/1048] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:29.850282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.851113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc99a39-02bf-46af-9cc1-c0785cc8671c None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:29.879893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.895171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:29.895171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:29.895559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:29.895912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:29.896638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] RBAC: Adding query filter params (type=image) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:29.897142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:29.899346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:29.905074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 509/1049] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/services?type=image => generated 350 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:29.911847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:29.939948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:29.974611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1690bc1c-b165-4258-9748-ccf827f815a9 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:30.014210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603d5417-6991-4462-89f1-b4dbcd01b795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:30.014840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d5417-6991-4462-89f1-b4dbcd01b795 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:30.015731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d5417-6991-4462-89f1-b4dbcd01b795 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:30.016166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d5417-6991-4462-89f1-b4dbcd01b795 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:30.016903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603d5417-6991-4462-89f1-b4dbcd01b795 None admin] RBAC: Adding query filter params (role.id=96eb47d9c9e341ab91ecffdb8ea813c8, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=40e12aa547b84ac38fd222a9f5f62ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:30.017208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603d5417-6991-4462-89f1-b4dbcd01b795 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:30.018710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603d5417-6991-4462-89f1-b4dbcd01b795 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:30.034448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 541/1050] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:29 2022] GET /identity/v3/role_assignments?user.id=40e12aa547b84ac38fd222a9f5f62ae2&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=96eb47d9c9e341ab91ecffdb8ea813c8 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:30.034663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068932b4-3fc5-444a-a82e-4d97de5a4022 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:30.067439 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4699b774-b297-43aa-85c2-facbe2dba3c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:30.068062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4699b774-b297-43aa-85c2-facbe2dba3c2 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:30.068341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4699b774-b297-43aa-85c2-facbe2dba3c2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:30.068614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4699b774-b297-43aa-85c2-facbe2dba3c2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:30.069306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4699b774-b297-43aa-85c2-facbe2dba3c2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:30.070715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4699b774-b297-43aa-85c2-facbe2dba3c2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:30.181796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 510/1051] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:34:29 2022] POST /identity/v3/endpoints => generated 327 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:34:31.040659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe8754-44f1-4397-ae2c-457fe1df1c7c None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:31.041214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe8754-44f1-4397-ae2c-457fe1df1c7c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:31.041214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe8754-44f1-4397-ae2c-457fe1df1c7c None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:31.042729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 542/1052] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:31.048911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9090bc-2ed0-4ffc-b100-b09f72b25bd1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:31.049285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9090bc-2ed0-4ffc-b100-b09f72b25bd1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:31.049454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9090bc-2ed0-4ffc-b100-b09f72b25bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:31.068279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-fe9090bc-2ed0-4ffc-b100-b09f72b25bd1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:31.069477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fe9090bc-2ed0-4ffc-b100-b09f72b25bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:31.108426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:31.149462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 511/1053] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:31.424649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebe8754-44f1-4397-ae2c-457fe1df1c7c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:31.529545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebe8754-44f1-4397-ae2c-457fe1df1c7c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:31.558970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:31.564585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:31.565217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:31.565496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:31.565769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:31.576029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:31.581469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:31.582365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:34:31.583251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:31.586760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 543/1054] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:34:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:31.588627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9090bc-2ed0-4ffc-b100-b09f72b25bd1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:31.693083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9090bc-2ed0-4ffc-b100-b09f72b25bd1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:31.727124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:31.727722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:31.727991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:31.728250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:31.728898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:31.729192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:31.730557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:31.743723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 512/1055] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:34:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:31.744974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:31.844946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ed5ff5-5014-4486-bc32-d64393a02422 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:31.878785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0e750b-4306-4e35-ab96-1644df5861ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:31.879409 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0e750b-4306-4e35-ab96-1644df5861ad None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:31.879683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0e750b-4306-4e35-ab96-1644df5861ad None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:31.879952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0e750b-4306-4e35-ab96-1644df5861ad None admin] PATH_INFO: `/v3/users/b4d67ff779fb44ae8475de4918909f0a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:31.894041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0e750b-4306-4e35-ab96-1644df5861ad None admin] RBAC: Authorizing `identity:get_user(user_id=b4d67ff779fb44ae8475de4918909f0a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:31.895511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0e750b-4306-4e35-ab96-1644df5861ad None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:31.897308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 544/1056] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:31 2022] GET /identity/v3/users/b4d67ff779fb44ae8475de4918909f0a => generated 285 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:31.903000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:31.934856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:32.006143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9565e66-7e75-4f1d-b2d9-65af0e3a3990 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:32.039210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:32.039814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.040088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.040355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.112717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:32.114163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:32.143717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:32.188472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4065f7-59a1-4232-86d5-90b4e4a531b2 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.189015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4065f7-59a1-4232-86d5-90b4e4a531b2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.189015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4065f7-59a1-4232-86d5-90b4e4a531b2 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.190580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 545/1057] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:32.195425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec8afde-3269-458f-8d88-9adbc82f0595 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.195708 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec8afde-3269-458f-8d88-9adbc82f0595 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.196173 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec8afde-3269-458f-8d88-9adbc82f0595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.213627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-fec8afde-3269-458f-8d88-9adbc82f0595 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:32.215055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fec8afde-3269-458f-8d88-9adbc82f0595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:32.221242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 513/1058] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:32.227884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:32.272835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:32.299888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 546/1059] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:32.338495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9739c077-4e5a-4096-a90d-c821898be500 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:32.368262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:32.379656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a341c4-89b2-4bb7-ad1a-6fdd3c5297bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:32.380299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a341c4-89b2-4bb7-ad1a-6fdd3c5297bb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.380571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a341c4-89b2-4bb7-ad1a-6fdd3c5297bb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.380837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a341c4-89b2-4bb7-ad1a-6fdd3c5297bb None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.381945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a341c4-89b2-4bb7-ad1a-6fdd3c5297bb None admin] RBAC: Authorizing `identity:get_project(project_id=9d5c45d32a88485cb6bc47b41684c7f8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:32.383409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a341c4-89b2-4bb7-ad1a-6fdd3c5297bb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:32.385182 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 514/1060] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:34:32 2022] GET /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8 => generated 311 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:32.391774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec8afde-3269-458f-8d88-9adbc82f0595 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:32.401702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd939405-53b9-42e8-bb44-41c46547e7b4 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.401984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd939405-53b9-42e8-bb44-41c46547e7b4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.402256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd939405-53b9-42e8-bb44-41c46547e7b4 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.403764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 515/1061] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:32.409222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95977d86-b7fb-41ac-a0d5-9dfd528c0054 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.409341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95977d86-b7fb-41ac-a0d5-9dfd528c0054 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.409539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95977d86-b7fb-41ac-a0d5-9dfd528c0054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.428324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-95977d86-b7fb-41ac-a0d5-9dfd528c0054 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:32.429505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-95977d86-b7fb-41ac-a0d5-9dfd528c0054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:32.508234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 516/1062] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:32.508798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec8afde-3269-458f-8d88-9adbc82f0595 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:32.544201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:32.545131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.545625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.546102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.547707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8, user.id=b4d67ff779fb44ae8475de4918909f0a) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:32.548069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:32.549504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:32.555127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 547/1063] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:32 2022] GET /identity/v3/role_assignments?user.id=b4d67ff779fb44ae8475de4918909f0a&scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:32.567206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95977d86-b7fb-41ac-a0d5-9dfd528c0054 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:32.681969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95977d86-b7fb-41ac-a0d5-9dfd528c0054 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:32.712958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:32.721746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:32.729162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.729162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.729162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.734971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:32.739792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:32.740700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:32.741523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 517/1064] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Oct 4 08:34:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:32.749332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:32.780237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:32.798052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:32.855061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eea62e-2844-4c0b-919e-3caa3fb1e8c3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:32.888576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad6273c-bef5-45b6-a452-a77f97c26609 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:32.894575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:32.895192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.895472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.895745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.896415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:32.897112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:32.898092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:32.912866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 548/1065] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Oct 4 08:34:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:32.913983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:32.928236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:32.928853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:32.929128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:32.929396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:32.990657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:33.005637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.007613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.021930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2141c6-21d0-44fd-a63f-4552b8848497 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.028974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:33.059877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.060702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.060986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.061425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.074285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.075792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:33.091974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:33.092947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:33.099044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:33.099044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:33.099044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:33.099044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 549/1066] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:33.099290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:33.124770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 518/1067] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:33.130857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:33.209008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339de0d-1d4e-4d92-abb7-22b75598f681 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.221367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:33.235129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a0ef0c-6d0a-4572-a483-32f2b3d386fb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.244627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.245243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.245519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.245791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.246456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:33.246764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.248129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.266292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 550/1068] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:33.268281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:33.277650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.278371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.278748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.279082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.287652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:33.292312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.293831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:33.308803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:33.309997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:33.311401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:33.311401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:33.311401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:33.311401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 519/1069] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:33.316923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:33.380011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80926f84-5948-4d92-af61-6c70a6ffd4f3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.423621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.424554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.425158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.425594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.432080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a187df8e-6f0f-416d-a83d-fd5eac9dd399 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.434338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.435975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:34:33.449801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:33.450857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:33.451794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 551/1070] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:33.456847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:33.472338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.472961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.473212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.473487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.474129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:33.474449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.475827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.493336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:33.496957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 520/1071] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:33.558497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:33.582305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866e25e3-4e8b-4d5c-a68b-94dc44f86237 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.619706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.620323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.620598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.620864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.621595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:33.621894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.623376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.648277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 552/1072] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:33.649884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:33.762539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1806b6df-6673-4ee9-af62-1e1ae0ce6b23 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.801382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.802022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.802301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.802602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.814178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.815670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:33.831434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:33.832425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:33.833318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:33.833318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:33.833318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:33.833318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 521/1073] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:33.839820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:33.937858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515efd7-c216-4a2a-a6cf-b769fea815a4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:33.970856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:33.971462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:33.971740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:33.972007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:33.972645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:33.972945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:33.974325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:33.976989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:33.986242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 553/1074] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:33.993875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:34.043148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:34.088821 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc308afb-f4bb-4b7e-a494-5f74e5676321 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:34.121796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:34.122426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:34.122714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:34.122979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:34.129545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:34.131004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:34.142461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:34.143354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:34.144218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:34.146542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 522/1075] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:34.149284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:34.247082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b519aff0-7809-4d73-90d4-af922cc0ad42 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:34.279903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595ad699-7453-4d44-8991-717b5a0311dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:34.280557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ad699-7453-4d44-8991-717b5a0311dd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:34.281347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ad699-7453-4d44-8991-717b5a0311dd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:34.281618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ad699-7453-4d44-8991-717b5a0311dd None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:34.282298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595ad699-7453-4d44-8991-717b5a0311dd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:34.282625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595ad699-7453-4d44-8991-717b5a0311dd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:34.284042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595ad699-7453-4d44-8991-717b5a0311dd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:34.295697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 554/1076] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:34.301675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:34.400145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a42bff-fc9e-4d2f-9d2f-ea212bd9572d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:34.413411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:34.433722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa019064-4eb3-49f7-a9c8-35d611ac2869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:34.434377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa019064-4eb3-49f7-a9c8-35d611ac2869 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:34.434685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa019064-4eb3-49f7-a9c8-35d611ac2869 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:34.434955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa019064-4eb3-49f7-a9c8-35d611ac2869 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/40e12aa547b84ac38fd222a9f5f62ae2/roles/96eb47d9c9e341ab91ecffdb8ea813c8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:34.437149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa019064-4eb3-49f7-a9c8-35d611ac2869 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=40e12aa547b84ac38fd222a9f5f62ae2, role_id=96eb47d9c9e341ab91ecffdb8ea813c8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:34.438775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa019064-4eb3-49f7-a9c8-35d611ac2869 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:34.506067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 523/1077] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:34:34 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/40e12aa547b84ac38fd222a9f5f62ae2/roles/96eb47d9c9e341ab91ecffdb8ea813c8 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:34:34.746723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3341da-2e68-403d-a73f-00ec034edf7b None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:34.747008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3341da-2e68-403d-a73f-00ec034edf7b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:34.747282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3341da-2e68-403d-a73f-00ec034edf7b None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:34.748641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 555/1078] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:34.753905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057921ff-7de2-4342-adf7-6df3f64abc3a None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:34.754256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057921ff-7de2-4342-adf7-6df3f64abc3a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:34.754621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057921ff-7de2-4342-adf7-6df3f64abc3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:34.773015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-057921ff-7de2-4342-adf7-6df3f64abc3a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:34.774195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-057921ff-7de2-4342-adf7-6df3f64abc3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:34.838739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 524/1079] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:35.113153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3341da-2e68-403d-a73f-00ec034edf7b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:35.170130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:35.214530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3341da-2e68-403d-a73f-00ec034edf7b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:35.248284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:35.248887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:35.249170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:35.249511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:35.259391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:35.264116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:35.265763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:34:35.267143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:35.271006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 556/1080] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:34:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:35.271980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057921ff-7de2-4342-adf7-6df3f64abc3a None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:35.369841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057921ff-7de2-4342-adf7-6df3f64abc3a None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:35.371487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:35.407449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:35.408050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:35.408325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:35.408594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:35.409251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:35.409552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:35.410978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:35.418894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 525/1081] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:34:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:35.424916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:35.526205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4893d983-e0fa-426e-8fa2-0844c0cf7252 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:35.542822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:35.560006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:35.560633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:35.560899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:35.561169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] PATH_INFO: `/v3/users/b4d67ff779fb44ae8475de4918909f0a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:35.563085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] RBAC: Authorizing `identity:get_user(user_id=b4d67ff779fb44ae8475de4918909f0a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:35.564514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:35.566074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 557/1082] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:35 2022] GET /identity/v3/users/b4d67ff779fb44ae8475de4918909f0a => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:35.572161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:35.689015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2ad499-b026-483c-b27a-3338ee379d98 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:35.721716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1d36ea-63f4-46fd-b9eb-5952fedfaeda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:35.722342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d36ea-63f4-46fd-b9eb-5952fedfaeda None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:35.722650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d36ea-63f4-46fd-b9eb-5952fedfaeda None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:35.722922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d36ea-63f4-46fd-b9eb-5952fedfaeda None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:35.724044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1d36ea-63f4-46fd-b9eb-5952fedfaeda None admin] RBAC: Authorizing `identity:get_project(project_id=9d5c45d32a88485cb6bc47b41684c7f8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:35.725464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1d36ea-63f4-46fd-b9eb-5952fedfaeda None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:35.727128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 526/1083] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:34:35 2022] GET /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8 => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:35.733014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:35.831523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af6763b-b64d-4684-b35a-8381d795552f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:35.864700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17597771-f227-4dd6-9c0b-c9aa8bed63e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:35.865346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17597771-f227-4dd6-9c0b-c9aa8bed63e0 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:35.865618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17597771-f227-4dd6-9c0b-c9aa8bed63e0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:35.865885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17597771-f227-4dd6-9c0b-c9aa8bed63e0 None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8/users/b4d67ff779fb44ae8475de4918909f0a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:35.867792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17597771-f227-4dd6-9c0b-c9aa8bed63e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d5c45d32a88485cb6bc47b41684c7f8, user_id=b4d67ff779fb44ae8475de4918909f0a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:35.869480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17597771-f227-4dd6-9c0b-c9aa8bed63e0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:35.946014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 558/1084] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:34:35 2022] PUT /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8/users/b4d67ff779fb44ae8475de4918909f0a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:34:36.707146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3507430d-6f0d-4a0b-8881-d76693bc3a09 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:36.707665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3507430d-6f0d-4a0b-8881-d76693bc3a09 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:36.707665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3507430d-6f0d-4a0b-8881-d76693bc3a09 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:36.708940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 527/1085] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:36.714085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc5846b-4f8a-4b5a-96c2-66ce72e35805 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:36.714496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc5846b-4f8a-4b5a-96c2-66ce72e35805 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:36.714667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc5846b-4f8a-4b5a-96c2-66ce72e35805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:36.732663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-5cc5846b-4f8a-4b5a-96c2-66ce72e35805 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:36.734420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5cc5846b-4f8a-4b5a-96c2-66ce72e35805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:36.773054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:36.799970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 559/1086] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:37.073016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3507430d-6f0d-4a0b-8881-d76693bc3a09 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:37.104099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:37.174689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3507430d-6f0d-4a0b-8881-d76693bc3a09 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:37.207596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:37.208216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:37.208489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:37.208751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:37.218459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:37.223088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:37.224103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:37.224941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 528/1087] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Oct 4 08:34:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:37.230057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc5846b-4f8a-4b5a-96c2-66ce72e35805 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:37.324933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc5846b-4f8a-4b5a-96c2-66ce72e35805 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:37.357193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:37.357790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:37.358062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:37.358329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:37.359549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:37.359842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:37.361196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:37.369156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 560/1088] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Oct 4 08:34:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:37.376206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:37.465250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:37.473785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cba8da-dc34-448a-8edb-8ca3e6d90c09 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:37.507738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:37.508377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:37.508649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:37.508911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:37.518803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:37.520226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:37.534917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:37.535886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:37.536773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:37.536773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:37.536773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:37.536773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 529/1089] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:37.541691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:37.575851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:37.638594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd628d-aa38-4cc4-81b1-ee9adb4fb6e9 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:37.671883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:37.672485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:37.672751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:37.673012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:37.673641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:37.673935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:37.675323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:37.694382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 561/1090] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:37.695575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:37.792267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e589730f-587e-4d57-a570-999bf9b533fb None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:37.824775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:37.825395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:37.825674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:37.825995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:37.834702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:37.836147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:34:37.847176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:37.852956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:37.853937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 530/1091] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Oct 4 08:34:37 2022] GET /identity/v3/users/nova => generated 82 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:37.854570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:37.948340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:37.954432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca991279-40d1-4f19-9eaa-67d75e4a5f3a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:37.988094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a90ae1-5ab7-4038-9a30-a1d5c4b7248d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:37.988721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a90ae1-5ab7-4038-9a30-a1d5c4b7248d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:37.988995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a90ae1-5ab7-4038-9a30-a1d5c4b7248d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:37.989260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a90ae1-5ab7-4038-9a30-a1d5c4b7248d None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:37.990004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a90ae1-5ab7-4038-9a30-a1d5c4b7248d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:37.990303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a90ae1-5ab7-4038-9a30-a1d5c4b7248d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:37.991855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a90ae1-5ab7-4038-9a30-a1d5c4b7248d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.013109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 562/1092] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Oct 4 08:34:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:38.019206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.023109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:38.114771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c4246c-01e1-46f7-8533-3cde18232d95 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.114970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c4246c-01e1-46f7-8533-3cde18232d95 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.115227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c4246c-01e1-46f7-8533-3cde18232d95 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.116532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 563/1093] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:34:38.118077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d18296-c7fb-4d7b-866b-3e291ca13fb4 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.121651 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62747072-334c-400e-bfd6-06f8bc579eba None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.121921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62747072-334c-400e-bfd6-06f8bc579eba None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.122184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62747072-334c-400e-bfd6-06f8bc579eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.139586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-62747072-334c-400e-bfd6-06f8bc579eba None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:38.141298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-62747072-334c-400e-bfd6-06f8bc579eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:38.151609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.152208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.152482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.152750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.205798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 564/1094] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:34:38.219125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:38.220568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.223171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:38.316657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 531/1095] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:38.322547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62747072-334c-400e-bfd6-06f8bc579eba None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.416505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:38.431523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62747072-334c-400e-bfd6-06f8bc579eba None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.465491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.466110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.466384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.466674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.476500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:38.477941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.478146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:34:38.492391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:38.493798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:34:38.495138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:34:38.495138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:34:38.495138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:38.495138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 565/1096] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:38.500393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.578955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932b1f9a-db25-4bda-b24e-db8bc6e5d595 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.581207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:38.598584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0232e1fd-1ccd-4209-9d39-169f9b159726 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.611782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.612408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.612682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.612949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.623354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:34:38.628688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:38.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:34:38.630520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 532/1097] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:34:38.636769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.636769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.636769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.636769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.637098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:38.637098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:38.637252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.637342 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.638970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:38.649224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 566/1098] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:38.654634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.735803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34120621-3b8d-47bd-a56a-b9d628d64d18 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.753317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56982a6-4f8d-48e2-a67a-b0bd9177e980 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.768547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.769140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.769416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.769681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.770329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:38.770652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:38.771992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.780313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 533/1099] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:38.786102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.786776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.787407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.787682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.787949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.794635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:38.796707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:34:38.807627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:34:38.808477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:34:38.809339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 567/1100] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:34:38.812952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:38.815177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.870672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:38.884490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5d2d7-52e1-4657-be9c-ae21b6943296 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.913252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7713969-0378-4b9f-8e85-96103d647e99 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:38.918108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.918763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.919036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.919299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] PATH_INFO: `/v3/users/b4d67ff779fb44ae8475de4918909f0a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.920414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] RBAC: Authorizing `identity:get_user(user_id=b4d67ff779fb44ae8475de4918909f0a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:38.922434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.928641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 534/1101] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/users/b4d67ff779fb44ae8475de4918909f0a => generated 285 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:38.929585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:38.947640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:38.948243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:38.948513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:38.948776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:38.949451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:38.949755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:38.951187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:38.962837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 568/1102] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:38.968913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:39.026200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:39.032624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e667c03c-22bb-4c2d-b718-4fc2966a3077 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:39.067187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b11c5b-8976-4ce8-8d77-971dd177e03d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:39.067593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:39.068205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:39.068486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:39.068763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:39.083747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:39.100876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:39.101476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:39.101758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:39.102028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:39.102777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] RBAC: Adding query filter params (role.id=96eb47d9c9e341ab91ecffdb8ea813c8, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=40e12aa547b84ac38fd222a9f5f62ae2) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:39.103080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:39.104520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:39.110557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 569/1103] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/role_assignments?user.id=40e12aa547b84ac38fd222a9f5f62ae2&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=96eb47d9c9e341ab91ecffdb8ea813c8 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:34:39.139694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:39.141820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:39.244055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:39.252569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 535/1104] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 08:34:39.258689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:39.360825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4854d51-1efc-4cf2-84a2-a72bb1338d0d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:39.395686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e810693-2a8c-4f52-80df-6761672ab5b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:39.396352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e810693-2a8c-4f52-80df-6761672ab5b9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:39.396628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e810693-2a8c-4f52-80df-6761672ab5b9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:39.396905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e810693-2a8c-4f52-80df-6761672ab5b9 None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:39.398013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e810693-2a8c-4f52-80df-6761672ab5b9 None admin] RBAC: Authorizing `identity:get_project(project_id=9d5c45d32a88485cb6bc47b41684c7f8)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:39.399480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e810693-2a8c-4f52-80df-6761672ab5b9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:39.401099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 570/1105] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:34:39 2022] GET /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:39.407806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:39.507930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d2fc7-d068-4b2c-947f-327f5430a219 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:39.549696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffc3e2a-e442-4242-98a5-15408607500c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:39.550311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffc3e2a-e442-4242-98a5-15408607500c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:39.550630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffc3e2a-e442-4242-98a5-15408607500c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:39.550911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffc3e2a-e442-4242-98a5-15408607500c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:39.551638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffc3e2a-e442-4242-98a5-15408607500c None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8, user.id=b4d67ff779fb44ae8475de4918909f0a) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:39.551944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffc3e2a-e442-4242-98a5-15408607500c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:39.553368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffc3e2a-e442-4242-98a5-15408607500c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:39.559885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 536/1106] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:39 2022] GET /identity/v3/role_assignments?user.id=b4d67ff779fb44ae8475de4918909f0a&scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:41.774881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4decf7a8-a6d6-4ade-af90-38c10a646c3c None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:41.775434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4decf7a8-a6d6-4ade-af90-38c10a646c3c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:41.775434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4decf7a8-a6d6-4ade-af90-38c10a646c3c None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:41.776742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 571/1107] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:41.782116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2192da-d6f6-408b-8247-29fba7861254 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:41.782505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2192da-d6f6-408b-8247-29fba7861254 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:41.782785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2192da-d6f6-408b-8247-29fba7861254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:41.800492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8a2192da-d6f6-408b-8247-29fba7861254 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:41.801671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8a2192da-d6f6-408b-8247-29fba7861254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:41.852583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:41.865313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 537/1108] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:42.137053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4decf7a8-a6d6-4ade-af90-38c10a646c3c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:42.187596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:42.239519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4decf7a8-a6d6-4ade-af90-38c10a646c3c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:42.273146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd25c935-9d85-4d74-aea2-04517296ec73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:42.273764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd25c935-9d85-4d74-aea2-04517296ec73 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:42.274041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd25c935-9d85-4d74-aea2-04517296ec73 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:42.274314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd25c935-9d85-4d74-aea2-04517296ec73 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:42.275467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd25c935-9d85-4d74-aea2-04517296ec73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:42.276896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd25c935-9d85-4d74-aea2-04517296ec73 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:42.278458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 572/1109] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:42.284360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2192da-d6f6-408b-8247-29fba7861254 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:42.380506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2192da-d6f6-408b-8247-29fba7861254 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:42.415285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983aa46a-f206-4a95-a4f7-a1dc6a260859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:42.415882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983aa46a-f206-4a95-a4f7-a1dc6a260859 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:42.416159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983aa46a-f206-4a95-a4f7-a1dc6a260859 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:42.416426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983aa46a-f206-4a95-a4f7-a1dc6a260859 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:42.417064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983aa46a-f206-4a95-a4f7-a1dc6a260859 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:42.418509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983aa46a-f206-4a95-a4f7-a1dc6a260859 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:42.527639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 538/1110] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:34:42 2022] POST /identity/v3/users => generated 286 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:44.724509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed75617-42a8-4749-a5a3-6258661216a6 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:44.725010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed75617-42a8-4749-a5a3-6258661216a6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:44.725010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed75617-42a8-4749-a5a3-6258661216a6 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:44.726301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 573/1111] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:44.731595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33e65d-f23e-4ed7-9ce4-e06881c29816 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:44.732013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33e65d-f23e-4ed7-9ce4-e06881c29816 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:44.732128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33e65d-f23e-4ed7-9ce4-e06881c29816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:44.751281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9d33e65d-f23e-4ed7-9ce4-e06881c29816 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:44.752453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9d33e65d-f23e-4ed7-9ce4-e06881c29816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:44.790934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:44.816629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 539/1112] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:45.083704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed75617-42a8-4749-a5a3-6258661216a6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:45.182602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed75617-42a8-4749-a5a3-6258661216a6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:45.197854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:45.214951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:45.215571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:45.215964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:45.216116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] PATH_INFO: `/v3/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:45.217495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] RBAC: Authorizing `identity:get_role(role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:45.219478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:45.220773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 574/1113] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:45 2022] GET /identity/v3/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:45.226719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33e65d-f23e-4ed7-9ce4-e06881c29816 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:45.321741 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33e65d-f23e-4ed7-9ce4-e06881c29816 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:45.356323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:45.356974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:45.357251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:45.357523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] PATH_INFO: `/v3/users/6930a84f4c4e4aab9faf4e909c5fd6f0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:45.371181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] RBAC: Authorizing `identity:get_user(user_id=6930a84f4c4e4aab9faf4e909c5fd6f0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:45.372639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:45.375421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 540/1114] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:45 2022] GET /identity/v3/users/6930a84f4c4e4aab9faf4e909c5fd6f0 => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:45.380266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:45.477700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86468ad-2c21-4943-a6a1-a303fa1ca094 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:45.509731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:45.510321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:45.510621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:45.510886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:45.559364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:45.574927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:45.576360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:45.674073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 575/1115] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:45.680028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:45.711995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:45.779488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92740c7d-fb8f-409d-8279-0853bb755d86 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:45.812998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8abe6a-3ccc-4da5-bcea-0e03824613e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:45.813628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8abe6a-3ccc-4da5-bcea-0e03824613e9 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:45.813905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8abe6a-3ccc-4da5-bcea-0e03824613e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:45.814175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8abe6a-3ccc-4da5-bcea-0e03824613e9 None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:45.815308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8abe6a-3ccc-4da5-bcea-0e03824613e9 None admin] RBAC: Authorizing `identity:get_project(project_id=9d5c45d32a88485cb6bc47b41684c7f8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:45.816716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8abe6a-3ccc-4da5-bcea-0e03824613e9 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:45.818328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 541/1116] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:34:45 2022] GET /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:45.824477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:45.903812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:45.918807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740e4f3-cbce-4df3-b301-e2ce1df980ef None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:45.952938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfabd1bc-1ed9-4ef6-b372-ec2908ae5574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:45.953462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabd1bc-1ed9-4ef6-b372-ec2908ae5574 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:45.953741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabd1bc-1ed9-4ef6-b372-ec2908ae5574 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:45.954013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabd1bc-1ed9-4ef6-b372-ec2908ae5574 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:45.954757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfabd1bc-1ed9-4ef6-b372-ec2908ae5574 None admin] RBAC: Adding query filter params (role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8, user.id=6930a84f4c4e4aab9faf4e909c5fd6f0) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:45.955062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfabd1bc-1ed9-4ef6-b372-ec2908ae5574 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:45.956494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfabd1bc-1ed9-4ef6-b372-ec2908ae5574 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:45.961630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 576/1117] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:45 2022] GET /identity/v3/role_assignments?user.id=6930a84f4c4e4aab9faf4e909c5fd6f0&scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:48.062105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ee2ce-1076-4930-86da-2f3a83156801 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:48.062669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ee2ce-1076-4930-86da-2f3a83156801 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:48.062669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ee2ce-1076-4930-86da-2f3a83156801 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:48.064001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 542/1118] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:48.069025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf8eb87-a7cf-48f7-a748-f5f6f16b4245 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:48.069441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf8eb87-a7cf-48f7-a748-f5f6f16b4245 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:48.069572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf8eb87-a7cf-48f7-a748-f5f6f16b4245 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:48.088032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7cf8eb87-a7cf-48f7-a748-f5f6f16b4245 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:48.089224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7cf8eb87-a7cf-48f7-a748-f5f6f16b4245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:48.153074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 577/1119] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:48.424216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5ee2ce-1076-4930-86da-2f3a83156801 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:48.518573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:48.522167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5ee2ce-1076-4930-86da-2f3a83156801 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:48.553575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:48.554186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:48.554487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:48.554754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] PATH_INFO: `/v3/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:48.556107 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] RBAC: Authorizing `identity:get_role(role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:48.557456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:48.558774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 543/1120] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:48 2022] GET /identity/v3/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:48.565260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf8eb87-a7cf-48f7-a748-f5f6f16b4245 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:48.642286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:48.663199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf8eb87-a7cf-48f7-a748-f5f6f16b4245 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:48.696954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:48.697570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:48.697843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:48.698107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] PATH_INFO: `/v3/users/6930a84f4c4e4aab9faf4e909c5fd6f0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:48.699234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] RBAC: Authorizing `identity:get_user(user_id=6930a84f4c4e4aab9faf4e909c5fd6f0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:48.700655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:48.702814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 578/1121] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:48 2022] GET /identity/v3/users/6930a84f4c4e4aab9faf4e909c5fd6f0 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:48.708376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:48.803364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe4e350-30e0-4de2-969e-d92fea3a7727 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:48.836780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf0b534-1335-4b2c-91a9-1369e72a9aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:48.837392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf0b534-1335-4b2c-91a9-1369e72a9aa7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:48.837667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf0b534-1335-4b2c-91a9-1369e72a9aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:48.837940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf0b534-1335-4b2c-91a9-1369e72a9aa7 None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:48.839057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf0b534-1335-4b2c-91a9-1369e72a9aa7 None admin] RBAC: Authorizing `identity:get_project(project_id=9d5c45d32a88485cb6bc47b41684c7f8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:48.840473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf0b534-1335-4b2c-91a9-1369e72a9aa7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:48.842065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 544/1122] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:34:48 2022] GET /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:48.847982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:48.945840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88eb10b-19f0-4140-b643-97e968cdf97f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:48.978098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df0ef99-0192-4f29-b7e5-cfaaf2728f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:48.978768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df0ef99-0192-4f29-b7e5-cfaaf2728f46 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:48.979048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df0ef99-0192-4f29-b7e5-cfaaf2728f46 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:48.979318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df0ef99-0192-4f29-b7e5-cfaaf2728f46 None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8/users/6930a84f4c4e4aab9faf4e909c5fd6f0/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:48.981461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df0ef99-0192-4f29-b7e5-cfaaf2728f46 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d5c45d32a88485cb6bc47b41684c7f8, user_id=6930a84f4c4e4aab9faf4e909c5fd6f0, role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:48.982912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df0ef99-0192-4f29-b7e5-cfaaf2728f46 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:49.041769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 579/1123] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:34:48 2022] PUT /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8/users/6930a84f4c4e4aab9faf4e909c5fd6f0/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:34:51.175873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe696a-dc80-4680-a3d4-040a92420814 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:51.176409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe696a-dc80-4680-a3d4-040a92420814 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:51.176409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe696a-dc80-4680-a3d4-040a92420814 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:51.177739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 545/1124] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:51.182938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e9c06-0d07-45a5-aa8e-7219e223780b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:51.183335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e9c06-0d07-45a5-aa8e-7219e223780b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:51.183491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e9c06-0d07-45a5-aa8e-7219e223780b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:51.195978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:51.203199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-673e9c06-0d07-45a5-aa8e-7219e223780b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:51.204381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-673e9c06-0d07-45a5-aa8e-7219e223780b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:51.269702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 580/1125] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:51.535202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbe696a-dc80-4680-a3d4-040a92420814 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:51.570364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:51.632974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbe696a-dc80-4680-a3d4-040a92420814 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:51.666658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:51.667273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:51.667545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:51.667807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] PATH_INFO: `/v3/roles/c8de71a7ff9248edb8beb59534bc30e6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:51.669167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] RBAC: Authorizing `identity:get_role(role_id=c8de71a7ff9248edb8beb59534bc30e6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:51.670539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:51.672591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 546/1126] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:51 2022] GET /identity/v3/roles/c8de71a7ff9248edb8beb59534bc30e6 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:51.678694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673e9c06-0d07-45a5-aa8e-7219e223780b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:51.776438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673e9c06-0d07-45a5-aa8e-7219e223780b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:51.810281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:51.810867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:51.811147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:51.811549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] PATH_INFO: `/v3/users/6930a84f4c4e4aab9faf4e909c5fd6f0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:51.812776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] RBAC: Authorizing `identity:get_user(user_id=6930a84f4c4e4aab9faf4e909c5fd6f0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:51.814215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:51.815840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 581/1127] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:34:51 2022] GET /identity/v3/users/6930a84f4c4e4aab9faf4e909c5fd6f0 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:51.821727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:51.922496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9709de-69cf-4a2c-b7c8-d18490ed5379 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:51.924756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:51.961021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:51.961622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:51.961898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:51.962170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:52.026588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:52.028018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:52.125205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 547/1128] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:34:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:34:52.131046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:52.134762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:52.226419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb3b8e1-dd7b-4630-bb17-46873e5a8d6f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:52.259189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b3fe6a-6715-4b1c-aaf9-85d866cbcd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:52.259804 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b3fe6a-6715-4b1c-aaf9-85d866cbcd80 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:52.260077 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b3fe6a-6715-4b1c-aaf9-85d866cbcd80 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:52.260344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b3fe6a-6715-4b1c-aaf9-85d866cbcd80 None admin] PATH_INFO: `/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:52.261436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b3fe6a-6715-4b1c-aaf9-85d866cbcd80 None admin] RBAC: Authorizing `identity:get_project(project_id=9d5c45d32a88485cb6bc47b41684c7f8)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:52.262883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b3fe6a-6715-4b1c-aaf9-85d866cbcd80 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:52.265654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 582/1129] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:34:52 2022] GET /identity/v3/projects/9d5c45d32a88485cb6bc47b41684c7f8 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:52.271444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:52.274506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:52.365079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a64dfb2-7124-40d2-aee9-feaaa2f844e7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:52.399359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1d85f9-095e-4d4d-adb1-810889e36cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:52.399960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1d85f9-095e-4d4d-adb1-810889e36cda None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:52.400237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1d85f9-095e-4d4d-adb1-810889e36cda None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:52.400508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1d85f9-095e-4d4d-adb1-810889e36cda None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:52.401226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1d85f9-095e-4d4d-adb1-810889e36cda None admin] RBAC: Adding query filter params (role.id=c8de71a7ff9248edb8beb59534bc30e6, scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8, user.id=6930a84f4c4e4aab9faf4e909c5fd6f0) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:34:52.401536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1d85f9-095e-4d4d-adb1-810889e36cda None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:52.403004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1d85f9-095e-4d4d-adb1-810889e36cda None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:52.408606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 548/1130] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:34:52 2022] GET /identity/v3/role_assignments?user.id=6930a84f4c4e4aab9faf4e909c5fd6f0&scope.project.id=9d5c45d32a88485cb6bc47b41684c7f8&role.id=c8de71a7ff9248edb8beb59534bc30e6 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:54.572492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45f19f6-4261-4bef-a07d-a8c699feed1a None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:54.573038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45f19f6-4261-4bef-a07d-a8c699feed1a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:54.573038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45f19f6-4261-4bef-a07d-a8c699feed1a None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:54.574343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 583/1131] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:54.579543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743802eb-0439-40e4-ad55-bdb26243e5cd None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:54.579911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743802eb-0439-40e4-ad55-bdb26243e5cd None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:54.580088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743802eb-0439-40e4-ad55-bdb26243e5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:54.598783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-743802eb-0439-40e4-ad55-bdb26243e5cd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:54.599964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-743802eb-0439-40e4-ad55-bdb26243e5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:54.659922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:54.664864 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 549/1132] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:54.930280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45f19f6-4261-4bef-a07d-a8c699feed1a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:54.999157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:55.028395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45f19f6-4261-4bef-a07d-a8c699feed1a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:55.060455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222341e9-4f94-46d4-a233-84ebdfd1d415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:55.061064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222341e9-4f94-46d4-a233-84ebdfd1d415 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:55.061376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222341e9-4f94-46d4-a233-84ebdfd1d415 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:55.061854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222341e9-4f94-46d4-a233-84ebdfd1d415 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:55.062777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222341e9-4f94-46d4-a233-84ebdfd1d415 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:55.064192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222341e9-4f94-46d4-a233-84ebdfd1d415 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:55.065707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 584/1133] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:55.071446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743802eb-0439-40e4-ad55-bdb26243e5cd None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:55.167457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743802eb-0439-40e4-ad55-bdb26243e5cd None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:55.200141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b23b6-6b77-469a-aaec-54286ccc43cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:55.200671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b23b6-6b77-469a-aaec-54286ccc43cf None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:55.200946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b23b6-6b77-469a-aaec-54286ccc43cf None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:55.201215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b23b6-6b77-469a-aaec-54286ccc43cf None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:55.202477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b23b6-6b77-469a-aaec-54286ccc43cf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:55.203961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b23b6-6b77-469a-aaec-54286ccc43cf None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:55.288671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 550/1134] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:34:55 2022] POST /identity/v3/projects => generated 311 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:34:57.476910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b4d27d-b615-49da-b52a-fa06f07d8c95 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:57.477433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b4d27d-b615-49da-b52a-fa06f07d8c95 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:57.477433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b4d27d-b615-49da-b52a-fa06f07d8c95 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:57.478807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 585/1135] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:34:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:34:57.484069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43264e67-b473-4af5-ab19-49bbc453d9b9 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:57.484515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43264e67-b473-4af5-ab19-49bbc453d9b9 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:57.484613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43264e67-b473-4af5-ab19-49bbc453d9b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:57.504160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-43264e67-b473-4af5-ab19-49bbc453d9b9 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:34:57.505344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-43264e67-b473-4af5-ab19-49bbc453d9b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:34:57.549546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:34:57.569553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 551/1136] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:34:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:34:57.839231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b4d27d-b615-49da-b52a-fa06f07d8c95 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:57.937587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b4d27d-b615-49da-b52a-fa06f07d8c95 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:57.970454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b17d00-ee99-454a-8580-88aeb6259201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:57.971072 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b17d00-ee99-454a-8580-88aeb6259201 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:57.971346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b17d00-ee99-454a-8580-88aeb6259201 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:57.971611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b17d00-ee99-454a-8580-88aeb6259201 None admin] PATH_INFO: `/v3/domains/default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:57.972698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b17d00-ee99-454a-8580-88aeb6259201 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:57.974108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b17d00-ee99-454a-8580-88aeb6259201 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:57.975674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 586/1137] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:34:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:34:57.981443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43264e67-b473-4af5-ab19-49bbc453d9b9 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:34:58.076509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43264e67-b473-4af5-ab19-49bbc453d9b9 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:34:58.113711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a85d21-0868-42f0-b5be-2be5207fa1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:34:58.114238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a85d21-0868-42f0-b5be-2be5207fa1ee None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:34:58.114545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a85d21-0868-42f0-b5be-2be5207fa1ee None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:34:58.114816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a85d21-0868-42f0-b5be-2be5207fa1ee None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:34:58.115479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a85d21-0868-42f0-b5be-2be5207fa1ee None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:34:58.116893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a85d21-0868-42f0-b5be-2be5207fa1ee None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:34:58.220320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 552/1138] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:34:57 2022] POST /identity/v3/users => generated 286 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:00.407444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd11c8c-1c7f-4d7f-a739-0670e559b36d None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:00.407904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd11c8c-1c7f-4d7f-a739-0670e559b36d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:00.408003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd11c8c-1c7f-4d7f-a739-0670e559b36d None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:00.409302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 587/1139] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:00.414588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd32687-6c6f-48cb-8834-e175cde90690 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:00.415006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd32687-6c6f-48cb-8834-e175cde90690 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:00.415131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd32687-6c6f-48cb-8834-e175cde90690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:00.432708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-cbd32687-6c6f-48cb-8834-e175cde90690 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:00.433874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cbd32687-6c6f-48cb-8834-e175cde90690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:00.462251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:00.497959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 553/1140] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:35:00.768168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd11c8c-1c7f-4d7f-a739-0670e559b36d None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:00.780056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:00.869213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd11c8c-1c7f-4d7f-a739-0670e559b36d None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:00.901585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:00.902198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:00.902502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:00.902778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:00.913670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:35:00.918343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:35:00.919531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:35:00.921101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 588/1141] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:35:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:35:00.926333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd32687-6c6f-48cb-8834-e175cde90690 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:01.020185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd32687-6c6f-48cb-8834-e175cde90690 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:01.052234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:01.052826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:01.053098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:01.053360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:01.054008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:01.054301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:01.055682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:01.069166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 554/1142] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:35:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:01.075621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:01.142769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:01.176132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38631af6-467d-42f4-b831-e61e7fe6ba3a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:01.208109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:01.208725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:01.209006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:01.209276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] PATH_INFO: `/v3/users/f25619395d204fa199e23225e6d43804` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:01.222525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] RBAC: Authorizing `identity:get_user(user_id=f25619395d204fa199e23225e6d43804)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:01.223966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:01.225533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 589/1143] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:35:01 2022] GET /identity/v3/users/f25619395d204fa199e23225e6d43804 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:01.231419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:01.260073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:01.331576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a3a056-6dd3-4a9c-ac6a-f95c79f8a1ad None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:01.364862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:01.365452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:01.365725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:01.365990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:01.436761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:01.438190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:01.464000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:01.533329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 555/1144] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:35:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 08:35:01.539385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:01.635011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835802d9-0e81-4c54-85cf-909c1255739d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:01.660328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:01.668648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b87a89c-ce01-4ce6-9e24-0d7199e33263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:01.669268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87a89c-ce01-4ce6-9e24-0d7199e33263 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:01.671220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87a89c-ce01-4ce6-9e24-0d7199e33263 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:01.671220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87a89c-ce01-4ce6-9e24-0d7199e33263 None admin] PATH_INFO: `/v3/projects/7c1cad0940954322a40635a72302d07a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:01.671220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b87a89c-ce01-4ce6-9e24-0d7199e33263 None admin] RBAC: Authorizing `identity:get_project(project_id=7c1cad0940954322a40635a72302d07a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:01.672502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b87a89c-ce01-4ce6-9e24-0d7199e33263 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:01.674091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 590/1145] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:35:01 2022] GET /identity/v3/projects/7c1cad0940954322a40635a72302d07a => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:01.680131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:01.777559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b543166-4a0c-4813-a1eb-aabdd5266059 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:01.802725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:01.809942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9bf49-5f51-4543-9e8d-e01c86932d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:01.810569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9bf49-5f51-4543-9e8d-e01c86932d5d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:01.810841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9bf49-5f51-4543-9e8d-e01c86932d5d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:01.811132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9bf49-5f51-4543-9e8d-e01c86932d5d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:01.811836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d9bf49-5f51-4543-9e8d-e01c86932d5d None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=7c1cad0940954322a40635a72302d07a, user.id=f25619395d204fa199e23225e6d43804) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:01.812134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d9bf49-5f51-4543-9e8d-e01c86932d5d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:01.813542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d9bf49-5f51-4543-9e8d-e01c86932d5d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:01.818611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 556/1146] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:35:01 2022] GET /identity/v3/role_assignments?user.id=f25619395d204fa199e23225e6d43804&scope.project.id=7c1cad0940954322a40635a72302d07a&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:03.940724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a6ff38-7502-48ad-a558-a18fabc405b6 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:03.941291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a6ff38-7502-48ad-a558-a18fabc405b6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:03.941291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a6ff38-7502-48ad-a558-a18fabc405b6 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:03.942952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 591/1147] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:03.948148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1693a0-30f6-46e8-91cd-7282a634f5d7 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:03.948586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1693a0-30f6-46e8-91cd-7282a634f5d7 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:03.948693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1693a0-30f6-46e8-91cd-7282a634f5d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:03.967133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9b1693a0-30f6-46e8-91cd-7282a634f5d7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:03.969129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9b1693a0-30f6-46e8-91cd-7282a634f5d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.032421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 557/1148] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:35:04.299888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a6ff38-7502-48ad-a558-a18fabc405b6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.398285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a6ff38-7502-48ad-a558-a18fabc405b6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.431880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.432501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.432780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.433052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.443894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:35:04.448623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:35:04.449515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:35:04.450609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 592/1149] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:35:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:35:04.456006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1693a0-30f6-46e8-91cd-7282a634f5d7 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.550711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1693a0-30f6-46e8-91cd-7282a634f5d7 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.571922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:04.583319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.583911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.584256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.584503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.585080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:04.585379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.586743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.595031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 558/1150] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:35:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:04.601691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.655777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:04.697749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183efc5f-ebf2-4f6a-aff0-4d5ed8729d36 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.730864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.731476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.731754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.732023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] PATH_INFO: `/v3/users/f25619395d204fa199e23225e6d43804` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.733338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] RBAC: Authorizing `identity:get_user(user_id=f25619395d204fa199e23225e6d43804)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.734844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.736397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 593/1151] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:35:04 2022] GET /identity/v3/users/f25619395d204fa199e23225e6d43804 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:04.742090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.836924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c403973f-43e3-4772-9173-43620879e9c4 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.868834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0c8087-ac6c-4003-beb3-7bd99cb50141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.869449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c8087-ac6c-4003-beb3-7bd99cb50141 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.869720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c8087-ac6c-4003-beb3-7bd99cb50141 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.869987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c8087-ac6c-4003-beb3-7bd99cb50141 None admin] PATH_INFO: `/v3/projects/7c1cad0940954322a40635a72302d07a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.871100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0c8087-ac6c-4003-beb3-7bd99cb50141 None admin] RBAC: Authorizing `identity:get_project(project_id=7c1cad0940954322a40635a72302d07a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.872503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0c8087-ac6c-4003-beb3-7bd99cb50141 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.874117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 559/1152] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:35:04 2022] GET /identity/v3/projects/7c1cad0940954322a40635a72302d07a => generated 311 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:04.881039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.977158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff3f9a8-c309-4dcd-abc5-7dd339105545 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.999734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:05.010573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba45963a-21ce-494b-a454-cfa793ac80e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.011213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45963a-21ce-494b-a454-cfa793ac80e0 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.011488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45963a-21ce-494b-a454-cfa793ac80e0 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.011752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45963a-21ce-494b-a454-cfa793ac80e0 None admin] PATH_INFO: `/v3/projects/7c1cad0940954322a40635a72302d07a/users/f25619395d204fa199e23225e6d43804/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.013603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba45963a-21ce-494b-a454-cfa793ac80e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c1cad0940954322a40635a72302d07a, user_id=f25619395d204fa199e23225e6d43804, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.015630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba45963a-21ce-494b-a454-cfa793ac80e0 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.128763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 594/1153] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:35:04 2022] PUT /identity/v3/projects/7c1cad0940954322a40635a72302d07a/users/f25619395d204fa199e23225e6d43804/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:07.243427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee72fb6-2445-49bd-8b32-ce523ac39448 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.244233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee72fb6-2445-49bd-8b32-ce523ac39448 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.244233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee72fb6-2445-49bd-8b32-ce523ac39448 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.245540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 560/1154] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:07.251346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e0872-73cb-4d32-aefe-b320c4ec6d95 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.251738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e0872-73cb-4d32-aefe-b320c4ec6d95 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.251886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e0872-73cb-4d32-aefe-b320c4ec6d95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.270298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6e5e0872-73cb-4d32-aefe-b320c4ec6d95 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:07.271861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6e5e0872-73cb-4d32-aefe-b320c4ec6d95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:07.335720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 595/1155] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:35:07.607768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee72fb6-2445-49bd-8b32-ce523ac39448 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.660697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:07.707151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee72fb6-2445-49bd-8b32-ce523ac39448 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.739949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.740555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.740867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.741135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.752529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:35:07.757242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:35:07.758421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:35:07.759281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 561/1156] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:35:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:35:07.764950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e0872-73cb-4d32-aefe-b320c4ec6d95 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.859138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e0872-73cb-4d32-aefe-b320c4ec6d95 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.871613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:07.892221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.892809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.893080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.893343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.893992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:07.894287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.895670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.903328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 596/1157] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:35:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:07.910059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.005945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775b8757-b3c4-4bbf-8eac-b6990ed13369 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.018910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:08.038598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.039213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.039482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.039744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] PATH_INFO: `/v3/users/f25619395d204fa199e23225e6d43804` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.040875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] RBAC: Authorizing `identity:get_user(user_id=f25619395d204fa199e23225e6d43804)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.042282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.043853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 562/1158] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:35:07 2022] GET /identity/v3/users/f25619395d204fa199e23225e6d43804 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:08.050182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.147053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d0420c-e05c-4da7-9249-5fe8a72fd43e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.179440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.180035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.180315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.180587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.221352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:08.245226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.246736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.341678 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 597/1159] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:35:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:35:08.347716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.444283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101a04d6-8848-4ca1-a583-d181065d5cf0 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.475555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e05955-13d0-4cda-b1f4-541174acf5ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.476168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e05955-13d0-4cda-b1f4-541174acf5ac None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.476442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e05955-13d0-4cda-b1f4-541174acf5ac None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.476708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e05955-13d0-4cda-b1f4-541174acf5ac None admin] PATH_INFO: `/v3/projects/7c1cad0940954322a40635a72302d07a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.477789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e05955-13d0-4cda-b1f4-541174acf5ac None admin] RBAC: Authorizing `identity:get_project(project_id=7c1cad0940954322a40635a72302d07a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.479229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e05955-13d0-4cda-b1f4-541174acf5ac None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.480842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 563/1160] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:35:08 2022] GET /identity/v3/projects/7c1cad0940954322a40635a72302d07a => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:08.487924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.561786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:08.585768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2325ae6-d7ff-41a2-b333-00960d02c4be None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.619498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7088d86-a798-42a4-ba44-41071c95987e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.620103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7088d86-a798-42a4-ba44-41071c95987e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.620381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7088d86-a798-42a4-ba44-41071c95987e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.620657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7088d86-a798-42a4-ba44-41071c95987e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.621369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7088d86-a798-42a4-ba44-41071c95987e None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=7c1cad0940954322a40635a72302d07a, user.id=f25619395d204fa199e23225e6d43804) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:08.621684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7088d86-a798-42a4-ba44-41071c95987e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.623134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7088d86-a798-42a4-ba44-41071c95987e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.629506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 598/1161] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:35:08 2022] GET /identity/v3/role_assignments?user.id=f25619395d204fa199e23225e6d43804&scope.project.id=7c1cad0940954322a40635a72302d07a&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:10.801031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8d8410-5581-4d93-8d59-25c3792dec39 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.801584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8d8410-5581-4d93-8d59-25c3792dec39 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.801584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8d8410-5581-4d93-8d59-25c3792dec39 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.802950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 564/1162] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:10.808268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce892f-29f1-4d19-87a7-017a43fa2b8a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.808680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce892f-29f1-4d19-87a7-017a43fa2b8a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.808788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ce892f-29f1-4d19-87a7-017a43fa2b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.828183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f1ce892f-29f1-4d19-87a7-017a43fa2b8a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:10.829352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f1ce892f-29f1-4d19-87a7-017a43fa2b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:10.892762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 599/1163] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:35:11.164263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8d8410-5581-4d93-8d59-25c3792dec39 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:11.213075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:11.263371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8d8410-5581-4d93-8d59-25c3792dec39 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:11.295768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:11.296356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:11.296629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:11.296893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:11.359984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:11.361392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:11.407145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:11.455986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 565/1164] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:35:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:35:11.461827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ce892f-29f1-4d19-87a7-017a43fa2b8a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:11.525348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:11.557084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ce892f-29f1-4d19-87a7-017a43fa2b8a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:11.589791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbeb267-5690-4c7a-afc1-c3ef3be28503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:11.590430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbeb267-5690-4c7a-afc1-c3ef3be28503 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:11.590718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbeb267-5690-4c7a-afc1-c3ef3be28503 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:11.590985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbeb267-5690-4c7a-afc1-c3ef3be28503 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:11.597959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbeb267-5690-4c7a-afc1-c3ef3be28503 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:11.599411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbeb267-5690-4c7a-afc1-c3ef3be28503 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0fbeb267-5690-4c7a-afc1-c3ef3be28503 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:35:11.610472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:35:11.611634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:35:11.612672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 600/1165] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:35:11 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:35:11.617768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:11.714089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed606ff-f45f-4ac4-bbe1-ba04e4b39f53 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:11.747851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab9703e-115e-4a60-ab2a-814c7b3f007e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:11.748977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9703e-115e-4a60-ab2a-814c7b3f007e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:11.749247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9703e-115e-4a60-ab2a-814c7b3f007e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:11.749512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9703e-115e-4a60-ab2a-814c7b3f007e None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:11.750155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab9703e-115e-4a60-ab2a-814c7b3f007e None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:11.750495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab9703e-115e-4a60-ab2a-814c7b3f007e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:11.751841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab9703e-115e-4a60-ab2a-814c7b3f007e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:11.758709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 566/1166] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Oct 4 08:35:11 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:13.872118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28eb9e0a-8b88-4902-ae38-be05a3de7dfb None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:13.872670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28eb9e0a-8b88-4902-ae38-be05a3de7dfb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:13.872670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28eb9e0a-8b88-4902-ae38-be05a3de7dfb None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:13.874234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 601/1167] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:13.879404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a02e53c-0653-4875-971b-edebdc55987e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:13.879816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a02e53c-0653-4875-971b-edebdc55987e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:13.879942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a02e53c-0653-4875-971b-edebdc55987e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:13.897969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-4a02e53c-0653-4875-971b-edebdc55987e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:13.899178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4a02e53c-0653-4875-971b-edebdc55987e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:13.906756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:13.963325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 567/1168] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:35:14.230054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28eb9e0a-8b88-4902-ae38-be05a3de7dfb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:14.333638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28eb9e0a-8b88-4902-ae38-be05a3de7dfb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:14.363276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:14.370977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8866ab1-9f7b-4e2c-8e37-00d209c37c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:14.371583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8866ab1-9f7b-4e2c-8e37-00d209c37c25 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:14.371862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8866ab1-9f7b-4e2c-8e37-00d209c37c25 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:14.372136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8866ab1-9f7b-4e2c-8e37-00d209c37c25 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:14.372819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8866ab1-9f7b-4e2c-8e37-00d209c37c25 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:14.375745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8866ab1-9f7b-4e2c-8e37-00d209c37c25 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:14.438050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 602/1169] 10.222.0.27 () {60 vars in 1138 bytes} [Tue Oct 4 08:35:14 2022] POST /identity/v3/domains => generated 267 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:16.644735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44db949a-523c-4fdf-8713-32b90329a88c None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:16.645232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44db949a-523c-4fdf-8713-32b90329a88c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:16.645232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44db949a-523c-4fdf-8713-32b90329a88c None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:16.646548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 568/1170] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:16.653027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f2e39-ca76-48d1-bccf-8c63e2437332 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:16.653426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f2e39-ca76-48d1-bccf-8c63e2437332 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:16.653573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f2e39-ca76-48d1-bccf-8c63e2437332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:16.672087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-092f2e39-ca76-48d1-bccf-8c63e2437332 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:16.673266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-092f2e39-ca76-48d1-bccf-8c63e2437332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:16.737380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 603/1171] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:35:17.008942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44db949a-523c-4fdf-8713-32b90329a88c None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:17.105111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44db949a-523c-4fdf-8713-32b90329a88c None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:17.137604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970448b6-f823-49df-a90f-4d111e22df8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:17.138219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970448b6-f823-49df-a90f-4d111e22df8c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:17.138521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970448b6-f823-49df-a90f-4d111e22df8c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:17.138910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970448b6-f823-49df-a90f-4d111e22df8c None admin] PATH_INFO: `/v3/domains/f2c8a263cf3541489803519d1f045a86` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:17.148830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970448b6-f823-49df-a90f-4d111e22df8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=f2c8a263cf3541489803519d1f045a86)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:17.150279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970448b6-f823-49df-a90f-4d111e22df8c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:17.151859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 569/1172] 10.222.0.27 () {56 vars in 1183 bytes} [Tue Oct 4 08:35:17 2022] GET /identity/v3/domains/f2c8a263cf3541489803519d1f045a86 => generated 267 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:17.158126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092f2e39-ca76-48d1-bccf-8c63e2437332 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:17.253918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092f2e39-ca76-48d1-bccf-8c63e2437332 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:17.270158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:17.286648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8ada55-ed2b-4002-9e2d-19fcd3270c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:17.287244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8ada55-ed2b-4002-9e2d-19fcd3270c9e None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:17.287520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8ada55-ed2b-4002-9e2d-19fcd3270c9e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:17.287789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8ada55-ed2b-4002-9e2d-19fcd3270c9e None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:17.288421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8ada55-ed2b-4002-9e2d-19fcd3270c9e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:17.289847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8ada55-ed2b-4002-9e2d-19fcd3270c9e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:17.379763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 604/1173] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:35:17 2022] POST /identity/v3/projects => generated 361 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:19.562758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3d6c3-de51-4b04-b103-06eeb4d78370 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.563306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3d6c3-de51-4b04-b103-06eeb4d78370 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.563306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3d6c3-de51-4b04-b103-06eeb4d78370 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.564625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 570/1174] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:19.569922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa03866-f1d0-4f0a-b6cd-87fdb52f15bc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.570312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa03866-f1d0-4f0a-b6cd-87fdb52f15bc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.570495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa03866-f1d0-4f0a-b6cd-87fdb52f15bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.589590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6fa03866-f1d0-4f0a-b6cd-87fdb52f15bc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:19.590793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6fa03866-f1d0-4f0a-b6cd-87fdb52f15bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:19.653929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 605/1175] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:35:19.923368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3d6c3-de51-4b04-b103-06eeb4d78370 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:19.926193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:20.021200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3d6c3-de51-4b04-b103-06eeb4d78370 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:20.053994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb88ee78-9fda-440a-8cfe-9884dfc1f884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:20.054637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb88ee78-9fda-440a-8cfe-9884dfc1f884 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.054914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb88ee78-9fda-440a-8cfe-9884dfc1f884 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.055183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb88ee78-9fda-440a-8cfe-9884dfc1f884 None admin] PATH_INFO: `/v3/domains/f2c8a263cf3541489803519d1f045a86` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.056276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb88ee78-9fda-440a-8cfe-9884dfc1f884 None admin] RBAC: Authorizing `identity:get_domain(domain_id=f2c8a263cf3541489803519d1f045a86)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:20.057691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb88ee78-9fda-440a-8cfe-9884dfc1f884 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.059234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 571/1176] 10.222.0.27 () {56 vars in 1183 bytes} [Tue Oct 4 08:35:19 2022] GET /identity/v3/domains/f2c8a263cf3541489803519d1f045a86 => generated 267 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:20.064936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa03866-f1d0-4f0a-b6cd-87fdb52f15bc None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:20.167287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:20.170929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa03866-f1d0-4f0a-b6cd-87fdb52f15bc None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:20.203374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b773ea9-6518-4f4a-a54b-688315453f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:20.203912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b773ea9-6518-4f4a-a54b-688315453f0e None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.204189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b773ea9-6518-4f4a-a54b-688315453f0e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.204456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b773ea9-6518-4f4a-a54b-688315453f0e None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.205091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b773ea9-6518-4f4a-a54b-688315453f0e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:20.206539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b773ea9-6518-4f4a-a54b-688315453f0e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.293084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 606/1177] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:35:20 2022] POST /identity/v3/users => generated 311 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:22.502626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c06fd-c3dc-4b3f-8e73-9cf92209a30b None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.503172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c06fd-c3dc-4b3f-8e73-9cf92209a30b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.503172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c06fd-c3dc-4b3f-8e73-9cf92209a30b None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.504489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 572/1178] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:22.509606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c497c2-24c2-4b9d-a9c9-4e51c9e0a67b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.510036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c497c2-24c2-4b9d-a9c9-4e51c9e0a67b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.510128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c497c2-24c2-4b9d-a9c9-4e51c9e0a67b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.527891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c1c497c2-24c2-4b9d-a9c9-4e51c9e0a67b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:22.529054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c1c497c2-24c2-4b9d-a9c9-4e51c9e0a67b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:22.592372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 607/1179] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:35:22.873331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314c06fd-c3dc-4b3f-8e73-9cf92209a30b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:22.945881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:22.976798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314c06fd-c3dc-4b3f-8e73-9cf92209a30b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:23.009289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:23.009903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:23.010174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:23.010463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:23.020295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:35:23.025306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:35:23.026542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:35:23.027566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 573/1180] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:35:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:35:23.033718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c497c2-24c2-4b9d-a9c9-4e51c9e0a67b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:23.130824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:23.147434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c497c2-24c2-4b9d-a9c9-4e51c9e0a67b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:23.189619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:23.190232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:23.190537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:23.190806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:23.191471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:23.191772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:23.193147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:23.203481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 608/1181] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:35:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:23.211582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:23.331957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3918109b-cbe9-4ca1-b5bd-fd6b381a57e4 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:23.367269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:23.378459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:23.379106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:23.379452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:23.379717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] PATH_INFO: `/v3/users/7f5c503bb35040d58ba55c0347b3b12d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:23.394670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] RBAC: Authorizing `identity:get_user(user_id=7f5c503bb35040d58ba55c0347b3b12d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:23.397383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:23.399709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 574/1182] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:35:23 2022] GET /identity/v3/users/7f5c503bb35040d58ba55c0347b3b12d => generated 311 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:23.405567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:23.546248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa4242-a7cd-4cb2-9d61-ef8f336b77c8 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:23.594109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:23.594744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:23.595021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:23.595290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:23.616281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:23.682810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:23.684177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:23.805145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 609/1183] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:35:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:35:23.813527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:23.911371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66d5880-4ae6-44db-8045-1aa89b469d38 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:23.944189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5d81e4-1259-4664-930e-6cfc32bc8073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:23.944736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d81e4-1259-4664-930e-6cfc32bc8073 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:23.945009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d81e4-1259-4664-930e-6cfc32bc8073 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:23.945272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d81e4-1259-4664-930e-6cfc32bc8073 None admin] PATH_INFO: `/v3/projects/1857fbe660894567a2c12533c53bf18b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:23.946358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5d81e4-1259-4664-930e-6cfc32bc8073 None admin] RBAC: Authorizing `identity:get_project(project_id=1857fbe660894567a2c12533c53bf18b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:23.947786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5d81e4-1259-4664-930e-6cfc32bc8073 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:23.949379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 575/1184] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:35:23 2022] GET /identity/v3/projects/1857fbe660894567a2c12533c53bf18b => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:23.955638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:24.002886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:24.052476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ae766-d326-4262-9c32-03798db5778f None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:24.085638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f44329e-3428-4727-8031-c7b76aab99ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:24.086231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f44329e-3428-4727-8031-c7b76aab99ac None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:24.086537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f44329e-3428-4727-8031-c7b76aab99ac None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:24.086806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f44329e-3428-4727-8031-c7b76aab99ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:24.087511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f44329e-3428-4727-8031-c7b76aab99ac None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1857fbe660894567a2c12533c53bf18b, user.id=7f5c503bb35040d58ba55c0347b3b12d) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:24.087808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f44329e-3428-4727-8031-c7b76aab99ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:24.089229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f44329e-3428-4727-8031-c7b76aab99ac None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:24.095372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 610/1185] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:35:23 2022] GET /identity/v3/role_assignments?user.id=7f5c503bb35040d58ba55c0347b3b12d&scope.project.id=1857fbe660894567a2c12533c53bf18b&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:26.227521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d69f0-41bc-4fc0-a535-4c0cf7b120a8 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.228052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d69f0-41bc-4fc0-a535-4c0cf7b120a8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.228052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d69f0-41bc-4fc0-a535-4c0cf7b120a8 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.229368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 576/1186] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:26.234610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad6f6b5-05dd-4766-81aa-28f3ccea6c63 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.234985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad6f6b5-05dd-4766-81aa-28f3ccea6c63 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.235162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad6f6b5-05dd-4766-81aa-28f3ccea6c63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.253694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0ad6f6b5-05dd-4766-81aa-28f3ccea6c63 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:26.254904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0ad6f6b5-05dd-4766-81aa-28f3ccea6c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:26.321414 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 611/1187] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:35:26.593281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d69f0-41bc-4fc0-a535-4c0cf7b120a8 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:26.648540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:26.691223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d69f0-41bc-4fc0-a535-4c0cf7b120a8 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:26.723024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:26.723628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.723896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.724156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.734255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:35:26.738978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:35:26.740019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:35:26.740845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 577/1188] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:35:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:35:26.746197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad6f6b5-05dd-4766-81aa-28f3ccea6c63 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:26.787182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:26.840877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad6f6b5-05dd-4766-81aa-28f3ccea6c63 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:26.873716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:26.874244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.874548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.874816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.875465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:26.875758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:26.877108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:26.884704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 612/1189] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:35:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:26.890662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:26.987010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b663a6-c93c-4b69-bdfe-984c56dd5a72 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:27.003002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:27.019686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:27.020301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.020576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.029636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] PATH_INFO: `/v3/users/7f5c503bb35040d58ba55c0347b3b12d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.029636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] RBAC: Authorizing `identity:get_user(user_id=7f5c503bb35040d58ba55c0347b3b12d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:27.029636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:27.029636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 578/1190] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:35:26 2022] GET /identity/v3/users/7f5c503bb35040d58ba55c0347b3b12d => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:27.031177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:27.127601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401d904b-50ca-46eb-87ff-caa16baf7cf2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:27.129782 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:27.161399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4feba6-519e-431f-a13c-9a60c2f9eb54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:27.162019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4feba6-519e-431f-a13c-9a60c2f9eb54 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.162297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4feba6-519e-431f-a13c-9a60c2f9eb54 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.162604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4feba6-519e-431f-a13c-9a60c2f9eb54 None admin] PATH_INFO: `/v3/projects/1857fbe660894567a2c12533c53bf18b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.163716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4feba6-519e-431f-a13c-9a60c2f9eb54 None admin] RBAC: Authorizing `identity:get_project(project_id=1857fbe660894567a2c12533c53bf18b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:27.165143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4feba6-519e-431f-a13c-9a60c2f9eb54 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:27.166768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 613/1191] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:35:27 2022] GET /identity/v3/projects/1857fbe660894567a2c12533c53bf18b => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:27.172759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:27.267732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8935f-f2fc-42f6-acf0-1e6c36d8ecb2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:27.300578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfc9b8d-8229-40a5-af45-ac58b1345028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:27.301227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfc9b8d-8229-40a5-af45-ac58b1345028 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.301501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfc9b8d-8229-40a5-af45-ac58b1345028 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.301768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfc9b8d-8229-40a5-af45-ac58b1345028 None admin] PATH_INFO: `/v3/projects/1857fbe660894567a2c12533c53bf18b/users/7f5c503bb35040d58ba55c0347b3b12d/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.303640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfc9b8d-8229-40a5-af45-ac58b1345028 None admin] RBAC: Authorizing `identity:create_grant(project_id=1857fbe660894567a2c12533c53bf18b, user_id=7f5c503bb35040d58ba55c0347b3b12d, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:27.305057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfc9b8d-8229-40a5-af45-ac58b1345028 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:27.391114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 579/1192] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:35:27 2022] PUT /identity/v3/projects/1857fbe660894567a2c12533c53bf18b/users/7f5c503bb35040d58ba55c0347b3b12d/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:29.505144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f8e179-50ce-41ac-9d93-47e02b6d9c26 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:29.505802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f8e179-50ce-41ac-9d93-47e02b6d9c26 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:29.505802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f8e179-50ce-41ac-9d93-47e02b6d9c26 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:29.507434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 614/1193] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:35:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:29.512742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d54bc9-0476-46db-be29-f26897cc4c17 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:29.513134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d54bc9-0476-46db-be29-f26897cc4c17 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:29.513307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d54bc9-0476-46db-be29-f26897cc4c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:29.531356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-64d54bc9-0476-46db-be29-f26897cc4c17 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:29.532538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-64d54bc9-0476-46db-be29-f26897cc4c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:29.572354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:29.597265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 580/1194] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:35:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:35:29.879518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f8e179-50ce-41ac-9d93-47e02b6d9c26 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:29.978833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f8e179-50ce-41ac-9d93-47e02b6d9c26 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.010890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.011507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.011781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.012046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.023834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:35:30.030279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:35:30.031463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:35:30.032537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 615/1195] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:35:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:35:30.037466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d54bc9-0476-46db-be29-f26897cc4c17 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.132146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d54bc9-0476-46db-be29-f26897cc4c17 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.165073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.165665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.165942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.166206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.166884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:30.167181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.168521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.176303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 581/1196] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:35:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:30.182484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.214517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:30.281200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c8e70e-58dc-469b-9469-074a9d33a05c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.312838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.313451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.313727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.313994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] PATH_INFO: `/v3/users/7f5c503bb35040d58ba55c0347b3b12d` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.315139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] RBAC: Authorizing `identity:get_user(user_id=7f5c503bb35040d58ba55c0347b3b12d)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.316563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.318111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 616/1197] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Oct 4 08:35:30 2022] GET /identity/v3/users/7f5c503bb35040d58ba55c0347b3b12d => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:30.323845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.363570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:30.427711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5047614f-c942-44f5-a447-e87b646cba62 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.460839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.461435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.461711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.461981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.525757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.527204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.558829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:30.620582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 582/1198] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:35:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:35:30.627124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.722621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:30.725379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff96c8-c9ac-41f1-a5b0-bf06b1f968ab None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.758216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12db79c-bdf4-4999-bb8a-d599a679d882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.758856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12db79c-bdf4-4999-bb8a-d599a679d882 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.759132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12db79c-bdf4-4999-bb8a-d599a679d882 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.759401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12db79c-bdf4-4999-bb8a-d599a679d882 None admin] PATH_INFO: `/v3/projects/1857fbe660894567a2c12533c53bf18b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.760498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12db79c-bdf4-4999-bb8a-d599a679d882 None admin] RBAC: Authorizing `identity:get_project(project_id=1857fbe660894567a2c12533c53bf18b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.761910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12db79c-bdf4-4999-bb8a-d599a679d882 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.764134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 617/1199] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Oct 4 08:35:30 2022] GET /identity/v3/projects/1857fbe660894567a2c12533c53bf18b => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:30.770388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.866699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2777dd2-e89d-45d2-80e5-9cdf550828d3 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.899063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3b6c20-cf55-41c3-95d8-1d16e3780f4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.899649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b6c20-cf55-41c3-95d8-1d16e3780f4e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.899925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b6c20-cf55-41c3-95d8-1d16e3780f4e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.900195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b6c20-cf55-41c3-95d8-1d16e3780f4e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.900904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b6c20-cf55-41c3-95d8-1d16e3780f4e None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1857fbe660894567a2c12533c53bf18b, user.id=7f5c503bb35040d58ba55c0347b3b12d) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:30.901207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b6c20-cf55-41c3-95d8-1d16e3780f4e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.902652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b6c20-cf55-41c3-95d8-1d16e3780f4e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.905481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:30.908584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 583/1200] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:35:30 2022] GET /identity/v3/role_assignments?user.id=7f5c503bb35040d58ba55c0347b3b12d&scope.project.id=1857fbe660894567a2c12533c53bf18b&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:40.756401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=16 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:43.699126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adda2a4d-997e-43d9-aab1-1353cd0ff17e None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.700348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adda2a4d-997e-43d9-aab1-1353cd0ff17e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.700348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adda2a4d-997e-43d9-aab1-1353cd0ff17e None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.701479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 618/1201] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:36:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:36:43.706666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de9a5c4-3984-4d94-9d7a-da3be1006128 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.707064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de9a5c4-3984-4d94-9d7a-da3be1006128 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.707242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de9a5c4-3984-4d94-9d7a-da3be1006128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.725334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5de9a5c4-3984-4d94-9d7a-da3be1006128 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:43.726564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5de9a5c4-3984-4d94-9d7a-da3be1006128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:43.792342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 584/1202] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:36:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:36:44.062645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adda2a4d-997e-43d9-aab1-1353cd0ff17e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.161456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adda2a4d-997e-43d9-aab1-1353cd0ff17e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.195219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:44.195847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.196126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.196401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.207078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:44.208519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:36:44.222489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:44.223672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:36:44.224920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:36:44.224920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:44.224920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:36:44.224920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 619/1203] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:44.229695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de9a5c4-3984-4d94-9d7a-da3be1006128 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.330185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de9a5c4-3984-4d94-9d7a-da3be1006128 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.358005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:44.363719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0644331-f35c-4ed3-9aa5-37f888d64831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:44.364320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0644331-f35c-4ed3-9aa5-37f888d64831 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.364598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0644331-f35c-4ed3-9aa5-37f888d64831 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.364868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0644331-f35c-4ed3-9aa5-37f888d64831 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.365500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0644331-f35c-4ed3-9aa5-37f888d64831 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:44.365799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0644331-f35c-4ed3-9aa5-37f888d64831 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:44.367179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0644331-f35c-4ed3-9aa5-37f888d64831 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:44.379173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 585/1204] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:36:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:44.384957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.437800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:44.483546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1add6942-7e93-4853-be0f-b7cb960f817c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.517195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5baa59-88bc-4980-b394-28cbca4e6ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:44.517794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5baa59-88bc-4980-b394-28cbca4e6ae1 None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.518119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5baa59-88bc-4980-b394-28cbca4e6ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.518386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5baa59-88bc-4980-b394-28cbca4e6ae1 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.519049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5baa59-88bc-4980-b394-28cbca4e6ae1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:44.520472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5baa59-88bc-4980-b394-28cbca4e6ae1 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:44.628458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 620/1205] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Oct 4 08:36:44 2022] POST /identity/v3/users => generated 251 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:46.815273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefeb9ff-c855-48ca-8a40-f72edba3b9d0 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.815804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefeb9ff-c855-48ca-8a40-f72edba3b9d0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.816098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefeb9ff-c855-48ca-8a40-f72edba3b9d0 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.817457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 586/1206] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:36:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:36:46.822569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc8cd7-5510-40f4-a482-7f1eb97cca10 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.822996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc8cd7-5510-40f4-a482-7f1eb97cca10 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.823168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc8cd7-5510-40f4-a482-7f1eb97cca10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.841863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1cbc8cd7-5510-40f4-a482-7f1eb97cca10 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:46.843078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1cbc8cd7-5510-40f4-a482-7f1eb97cca10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:46.913394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 621/1207] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:36:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:36:47.184780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefeb9ff-c855-48ca-8a40-f72edba3b9d0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.293915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefeb9ff-c855-48ca-8a40-f72edba3b9d0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.328519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.329142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.329416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.329685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.339911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:36:47.345529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:47.346734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:47.347603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 587/1208] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:47.352828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc8cd7-5510-40f4-a482-7f1eb97cca10 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.392705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:47.457036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc8cd7-5510-40f4-a482-7f1eb97cca10 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.490894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.491497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.491776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.492046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.492697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:47.492994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.494653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.503466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 622/1209] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:47.509514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.558523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:47.615015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31145b06-8ff8-4bd2-b901-37c0ba4f9006 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.648839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.649457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.649767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.650002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.660424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.661876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:36:47.676339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:47.677441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:36:47.678332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:36:47.678332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:47.678332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:47.678332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 588/1210] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:47.683312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.780982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:47.783908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2827a80-9831-4bb8-ad3b-04fcd3242ece None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.816648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.817242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.817513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.817777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.818418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:47.818730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.820080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.831291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 623/1211] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:47.837099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.944442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660354bd-9780-4afc-9dd1-c846fbe0bf97 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.946327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:47.977756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.978377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.978679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.978941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.986554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.988026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:36:48.000349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:48.001501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:48.002669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 589/1212] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/users/placement => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:48.007331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.105028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e0c04-7a1d-4c0f-a801-cfff019c6ed2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.141186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.143937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.143937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.143937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.143937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:48.143937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.144724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.161990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 624/1213] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Oct 4 08:36:48 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:48.167756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.267563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d13cc5c-ad86-43e2-8019-0c34b1a3c220 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.301145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.301748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.302028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.302299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.331045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:48.369222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.370600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.463623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 590/1214] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:36:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:36:48.470341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.489405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:48.567534 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab2e72-f036-406f-b009-186965bdddc7 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.601756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.602545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.602823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.603114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.614060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.615527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:36:48.630272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:48.631399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:36:48.632453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:36:48.632453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:48.632453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:36:48.632453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 625/1215] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:48.638047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.702021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:48.738897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c89ce-ec94-451e-bcd9-06c82d46f125 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.772520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.773119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.773396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.773668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.774306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:48.774635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.776003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.788201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 591/1216] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:36:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:48.794199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.862130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:48.893128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85214446-0dc2-4915-aa01-9be93b699dab None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.927373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.927991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.928269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.928538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.935471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.936909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:36:48.949036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:48.951121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:36:48.952053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 626/1217] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:36:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:48.956615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:49.054628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cccf1-b58e-4a88-a0fc-3c1934785e19 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:49.080952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:49.092105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257dfdf7-1c68-411e-bdfd-08a32a547463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:49.092708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257dfdf7-1c68-411e-bdfd-08a32a547463 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.092981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257dfdf7-1c68-411e-bdfd-08a32a547463 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.093247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257dfdf7-1c68-411e-bdfd-08a32a547463 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.093927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257dfdf7-1c68-411e-bdfd-08a32a547463 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:49.094228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257dfdf7-1c68-411e-bdfd-08a32a547463 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:49.095726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257dfdf7-1c68-411e-bdfd-08a32a547463 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.106955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 592/1218] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:36:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:49.113152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:49.211113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209eecfc-1848-45b9-927a-688ed547043d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:49.238181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:49.244337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf2c12b-b7b9-419e-b76a-5eb2ed1ef027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:49.244969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2c12b-b7b9-419e-b76a-5eb2ed1ef027 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.245245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2c12b-b7b9-419e-b76a-5eb2ed1ef027 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.245513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2c12b-b7b9-419e-b76a-5eb2ed1ef027 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.246779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf2c12b-b7b9-419e-b76a-5eb2ed1ef027 None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=659e82e28da54e28a58d9d6e9c5e4215) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:49.247085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf2c12b-b7b9-419e-b76a-5eb2ed1ef027 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:49.248509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf2c12b-b7b9-419e-b76a-5eb2ed1ef027 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.258112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 627/1219] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:36:49 2022] GET /identity/v3/role_assignments?user.id=659e82e28da54e28a58d9d6e9c5e4215&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:51.391265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a7b5f-bcb6-47df-b0d3-28a8fab7e8ba None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:51.391813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a7b5f-bcb6-47df-b0d3-28a8fab7e8ba None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:51.391813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a7b5f-bcb6-47df-b0d3-28a8fab7e8ba None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:51.393118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 593/1220] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:36:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:36:51.398499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c580286c-88ab-47ba-8654-b3d23bd9424b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:51.398893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c580286c-88ab-47ba-8654-b3d23bd9424b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:51.399051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c580286c-88ab-47ba-8654-b3d23bd9424b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:51.417790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c580286c-88ab-47ba-8654-b3d23bd9424b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:51.419687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c580286c-88ab-47ba-8654-b3d23bd9424b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:51.489385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 628/1221] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:36:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:36:51.773563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15a7b5f-bcb6-47df-b0d3-28a8fab7e8ba None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:51.876963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15a7b5f-bcb6-47df-b0d3-28a8fab7e8ba None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:51.910023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:51.910670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:51.910945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:51.911214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] PATH_INFO: `/v3/roles/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:51.921370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:36:51.926108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:51.929959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:51.931031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 594/1222] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:36:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:51.934308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c580286c-88ab-47ba-8654-b3d23bd9424b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.029680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c580286c-88ab-47ba-8654-b3d23bd9424b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.052376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:52.063166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:52.063776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:52.064056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:52.064324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:52.064978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] RBAC: Adding query filter params (name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:52.065278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:52.066659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:52.074296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 629/1223] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:36:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:52.081155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.124594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:52.177552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a2307-ef54-4236-a1ba-14003533d111 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.210984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:52.211607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:52.211884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:52.212152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:52.222715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:52.224323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:36:52.239102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:52.245531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:36:52.247808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:36:52.247808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:52.247808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:52.247808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 595/1224] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:52.248084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.343589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59efa14-d92a-4d55-a817-b76b94168dab None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.377992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:52.378628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:52.378906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:52.379174 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:52.379819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:52.380120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:52.381480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:52.393398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 630/1225] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:36:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:52.399113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.491647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:52.499128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7e7165-d7c8-4c51-88a4-6038aee64058 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.530982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:52.531591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:52.531866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:52.532134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:52.538801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:52.540245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:36:52.552767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:52.558485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:52.560399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 596/1226] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:52 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:52.561170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.607963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:52.661677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed860bc-bf09-4a99-9de1-740ce4b66a18 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.695062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:52.695661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:52.695933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:52.696198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:52.696910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:52.697205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:52.698656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:52.717943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 631/1227] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Oct 4 08:36:52 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:36:52.723812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.821143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f6459f-f270-45ed-9009-e7d7a2163bb2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.854716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:52.856302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:52.856687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:52.857004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:52.869680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:52.871122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:36:52.884712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:52.885832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:36:52.887157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:36:52.887157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:52.887157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:52.887157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 597/1228] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:52.891883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.995223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aa34eb-9a28-4bce-825d-2b05b209d68a None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.997601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:53.029599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:53.030229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:53.030803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:53.031069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:53.031696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:53.031990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:53.033336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:53.044809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 632/1229] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:36:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:36:53.050631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:53.059948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:53.149834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4448f5-f133-4fc0-ab37-4314638f9453 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:53.182337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:53.182988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:53.183245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:53.183509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:53.189947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:53.191401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:36:53.202940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:53.205537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:53.206643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 598/1230] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:36:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:53.209871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:53.308975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c5f27-71eb-4354-9a4e-c87fa1b52ce3 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:53.342210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2a61df-3fa5-4f4e-9707-925f25da4527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:53.342837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2a61df-3fa5-4f4e-9707-925f25da4527 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:53.343112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2a61df-3fa5-4f4e-9707-925f25da4527 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:53.343375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2a61df-3fa5-4f4e-9707-925f25da4527 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:53.344052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2a61df-3fa5-4f4e-9707-925f25da4527 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:53.344353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2a61df-3fa5-4f4e-9707-925f25da4527 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:53.345755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2a61df-3fa5-4f4e-9707-925f25da4527 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:53.358060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 633/1231] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:36:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:53.364151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:53.429270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:53.463141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b84bf-670c-42e8-a6e3-9821c0fb191b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:53.495462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d16451a-d3f3-4d48-9ed5-78533ae75216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:53.496652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d16451a-d3f3-4d48-9ed5-78533ae75216 None admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:53.496925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d16451a-d3f3-4d48-9ed5-78533ae75216 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:53.497197 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d16451a-d3f3-4d48-9ed5-78533ae75216 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/659e82e28da54e28a58d9d6e9c5e4215/roles/4b9e5859f89942e79cfb0e553769100b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:53.511379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d16451a-d3f3-4d48-9ed5-78533ae75216 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=659e82e28da54e28a58d9d6e9c5e4215, role_id=4b9e5859f89942e79cfb0e553769100b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:53.513379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d16451a-d3f3-4d48-9ed5-78533ae75216 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:53.609811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 599/1232] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:36:53 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/659e82e28da54e28a58d9d6e9c5e4215/roles/4b9e5859f89942e79cfb0e553769100b => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:55.753084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f012f96-813e-4d83-a8c0-372e86831ebb None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.753633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f012f96-813e-4d83-a8c0-372e86831ebb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.753633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f012f96-813e-4d83-a8c0-372e86831ebb None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.755011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 634/1233] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:36:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:36:55.760598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143800b4-b759-4aa3-953d-371a7b3cd088 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.765829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143800b4-b759-4aa3-953d-371a7b3cd088 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.765829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143800b4-b759-4aa3-953d-371a7b3cd088 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.779790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-143800b4-b759-4aa3-953d-371a7b3cd088 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:55.781066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-143800b4-b759-4aa3-953d-371a7b3cd088 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.847375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 600/1234] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:36:56.123282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f012f96-813e-4d83-a8c0-372e86831ebb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.148345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:56.222737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f012f96-813e-4d83-a8c0-372e86831ebb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.256385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.257002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.257278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.257546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] PATH_INFO: `/v3/roles/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.267993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:36:56.272844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:56.273976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:36:56.275026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:36:56.279780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 635/1235] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:56.280788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143800b4-b759-4aa3-953d-371a7b3cd088 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.331247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:56.382535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143800b4-b759-4aa3-953d-371a7b3cd088 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.416587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.417191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.417466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.417734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.418472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] RBAC: Adding query filter params (name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:56.418774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.420168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.428061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 601/1236] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:36:56.434990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.525998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:56.546460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf1365-95e5-4f8a-a10b-b4e48914993b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.580102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.580716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.580998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.581265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.591460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.592923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:36:56.606490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:56.607567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:36:56.608839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:36:56.608839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:56.608839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:36:56.608839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 636/1237] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:56.613391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.713491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610cca19-8b83-4d52-b31c-846889cf11e8 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.722305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:56.747235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.747877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.748150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.748414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.749080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:56.749376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.750834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.762876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 602/1238] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:56.768669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.868126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eec5f-0f6e-49a2-af9b-fa5bf4a5198c None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.901687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.902397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.902787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.903000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.905631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:56.910421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.911912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:36:56.923386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:56.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:36:56.925429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 637/1239] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:56.930421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.029570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fa9469-5af1-45de-94b8-23051467938d None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:57.063487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:57.064087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.064367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.064633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.065347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:57.065646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:57.067157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:57.083948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 603/1240] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:57.089928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.188716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca23789-461d-4e13-a4ac-1c4fabc93cd4 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:57.222093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:57.222737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.223013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.223281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.283288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:57.289642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:57.291179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:57.386770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 638/1241] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:36:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:36:57.392715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.419836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:57.488257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7936f4b-d0f8-4eb7-9b54-7f81362a3f06 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:57.520862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:57.521477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.521747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.522012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.533157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:57.534678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:36:57.549671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:57.551231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:36:57.552189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:36:57.552189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:36:57.552189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:57.552189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 604/1242] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:36:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:57.556692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.651073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:57.657405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c4fd9-27db-44fb-a761-61a0a06a27db None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:57.689831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:57.690463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.690744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.691012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.691647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:57.691946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:57.693308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:57.704658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 639/1243] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:36:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:57.710682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.792692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:57.813972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5187375c-584e-400c-ade6-1429770d3767 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:57.847231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:57.847847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.848125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.848396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.855086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:57.856899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:36:57.868090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:36:57.869395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:36:57.870337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 605/1244] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:36:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:36:57.875937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.975871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a9474-f146-46e2-9677-ff713f69d2f2 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.008970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5025f0ac-5a6e-429e-bb71-5b90d4777a07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.009570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025f0ac-5a6e-429e-bb71-5b90d4777a07 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.009897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025f0ac-5a6e-429e-bb71-5b90d4777a07 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.010166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025f0ac-5a6e-429e-bb71-5b90d4777a07 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.010882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5025f0ac-5a6e-429e-bb71-5b90d4777a07 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:58.011182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5025f0ac-5a6e-429e-bb71-5b90d4777a07 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.013318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5025f0ac-5a6e-429e-bb71-5b90d4777a07 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.024923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 640/1245] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:36:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:58.031422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:58.132731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70abab0e-35ac-4642-9042-143d3f4f87c7 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.165386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc0483a-00d9-4a36-b54c-b784d891eaab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.165978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc0483a-00d9-4a36-b54c-b784d891eaab None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.166260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc0483a-00d9-4a36-b54c-b784d891eaab None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.166560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc0483a-00d9-4a36-b54c-b784d891eaab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.167270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc0483a-00d9-4a36-b54c-b784d891eaab None admin] RBAC: Adding query filter params (role.id=4b9e5859f89942e79cfb0e553769100b, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=659e82e28da54e28a58d9d6e9c5e4215) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:58.167571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc0483a-00d9-4a36-b54c-b784d891eaab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.169016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc0483a-00d9-4a36-b54c-b784d891eaab None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.174575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 606/1246] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:36:58 2022] GET /identity/v3/role_assignments?user.id=659e82e28da54e28a58d9d6e9c5e4215&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=4b9e5859f89942e79cfb0e553769100b => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:37:00.430591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9ce8e-8873-4ae4-9439-de09e1985b03 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.431187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9ce8e-8873-4ae4-9439-de09e1985b03 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.431187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9ce8e-8873-4ae4-9439-de09e1985b03 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.432506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 641/1247] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:37:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:37:00.437855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7a68b1-4c6e-4296-b118-006b41fc5795 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.438360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7a68b1-4c6e-4296-b118-006b41fc5795 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.438506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7a68b1-4c6e-4296-b118-006b41fc5795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.444748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:00.457120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8b7a68b1-4c6e-4296-b118-006b41fc5795 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:00.458313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8b7a68b1-4c6e-4296-b118-006b41fc5795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:00.526779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 607/1248] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:37:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:37:00.803903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da9ce8e-8873-4ae4-9439-de09e1985b03 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:00.821746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:00.903944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da9ce8e-8873-4ae4-9439-de09e1985b03 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:00.938861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:00.939476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.939750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.940014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.950728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:37:00.955499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:00.962357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:00.963369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 642/1249] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:37:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:00.964104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7a68b1-4c6e-4296-b118-006b41fc5795 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:01.069755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7a68b1-4c6e-4296-b118-006b41fc5795 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:01.098546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:01.103355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:01.103957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:01.104234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:01.104501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:01.105154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:01.105565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:01.106939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:01.114598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 608/1250] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:37:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:01.120677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:01.196163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:01.219354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e02c7b6-d911-444a-896a-0a684a19d75b None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:01.253828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:01.254466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:01.254742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:01.255005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:01.265370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:01.266848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:37:01.281283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:01.282418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:37:01.283351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:37:01.283351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:01.283351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:01.283351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 643/1251] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:01.288264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:01.387345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f0c1b-5322-43c7-abbe-1997f89ac5e2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:01.420348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:01.420937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:01.421210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:01.421475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:01.422102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:01.422399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:01.423761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:01.436243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 609/1252] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:37:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:37:01.441963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:01.540998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6072827-cefa-4e05-9df2-46b59bf009f6 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:01.567388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:01.575083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:01.575743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:01.576019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:01.576309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:01.583375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:01.584805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:37:01.595925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:01.611188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:01.612362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 644/1253] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:01.612990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:01.658222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:01.700200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104eb19-92f1-4ffb-a0ed-1c82b19ac4ab None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:01.735644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:01.736271 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:01.736546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:01.736813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:01.737534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:01.737835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:01.739315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:01.756799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 610/1254] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Oct 4 08:37:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:01.762786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:01.863207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35343bf8-b8ee-4fe7-bf88-b21781082071 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:01.895584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:01.897717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:01.897717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:01.897717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:01.952606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:01.964265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:01.966229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:02.070128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 645/1255] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:37:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:37:02.076693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:02.173725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c74be-1c53-47a6-a7c5-6bfd46a50755 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:02.187219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:02.207758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:02.208311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:02.208623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:02.208900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:02.219298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:02.220773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:37:02.235137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:02.236237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:37:02.237110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:37:02.237110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:02.237110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:02.237110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 611/1256] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:02.242015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:02.327051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:02.338317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ca55a-455c-42e2-a43b-2f72573db686 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:02.376672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:02.377290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:02.377562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:02.377829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:02.378496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:02.378793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:02.380149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:02.392069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 646/1257] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:37:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:02.397778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:02.498106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64657194-62e6-4c1d-a792-1a8a5dd3443a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:02.530560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:02.531185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:02.531488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:02.531762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] PATH_INFO: `/v3/projects/service` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:02.538225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:02.539687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:37:02.551005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:02.552310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:02.553748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 612/1258] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:37:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:02.558112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:02.659484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892b9fb8-f388-4be5-9a91-200955321d0d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:02.693898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3944aa-1a21-446b-93e5-dbc932c765d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:02.694526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3944aa-1a21-446b-93e5-dbc932c765d9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:02.694802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3944aa-1a21-446b-93e5-dbc932c765d9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:02.695067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3944aa-1a21-446b-93e5-dbc932c765d9 None admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:02.695744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3944aa-1a21-446b-93e5-dbc932c765d9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:02.696042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3944aa-1a21-446b-93e5-dbc932c765d9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:02.697450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3944aa-1a21-446b-93e5-dbc932c765d9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:02.710122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 647/1259] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:37:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:02.717362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:02.743412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:02.822748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91ba75c-92dc-4df8-8349-dba8797f622e None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:02.857179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cb7384-6380-4659-99c5-50651084bb18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:02.857920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb7384-6380-4659-99c5-50651084bb18 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:02.858201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb7384-6380-4659-99c5-50651084bb18 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:02.858506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb7384-6380-4659-99c5-50651084bb18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:02.859229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cb7384-6380-4659-99c5-50651084bb18 None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=659e82e28da54e28a58d9d6e9c5e4215) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:02.859538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cb7384-6380-4659-99c5-50651084bb18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:02.860962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cb7384-6380-4659-99c5-50651084bb18 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:02.866370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 613/1260] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:37:02 2022] GET /identity/v3/role_assignments?user.id=659e82e28da54e28a58d9d6e9c5e4215&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=1922dc727b87489999cc8916eacc229e => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:05.004687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ff760d-2a9e-49fa-8e49-9eb3422b29ba None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.005215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ff760d-2a9e-49fa-8e49-9eb3422b29ba None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.005215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ff760d-2a9e-49fa-8e49-9eb3422b29ba None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.006575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 648/1261] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:37:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:37:05.011645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866faf99-be3e-4835-a508-b3c2b8e5939b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.012050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866faf99-be3e-4835-a508-b3c2b8e5939b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.012294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866faf99-be3e-4835-a508-b3c2b8e5939b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.030815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-866faf99-be3e-4835-a508-b3c2b8e5939b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:05.032006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-866faf99-be3e-4835-a508-b3c2b8e5939b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:05.097889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 614/1262] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:37:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:37:05.374736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ff760d-2a9e-49fa-8e49-9eb3422b29ba None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:05.385365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:05.475600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ff760d-2a9e-49fa-8e49-9eb3422b29ba None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:05.508811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:05.509430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.509709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.509978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.520257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:37:05.525028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:05.526109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:05.527013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 649/1263] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:37:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:05.532322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866faf99-be3e-4835-a508-b3c2b8e5939b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:05.549871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:05.631198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866faf99-be3e-4835-a508-b3c2b8e5939b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:05.665796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:05.666393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.666802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.667070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.667726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] RBAC: Adding query filter params (name=admin) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:05.668016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:05.669385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:05.677218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 615/1264] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:37:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:05.683057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:05.750915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:05.784775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07f4b76-691c-47c8-9159-71857f9af471 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:05.818492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:05.819116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.819395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.819666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.829882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:05.831351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:37:05.846954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:05.848068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:37:05.849190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:37:05.849190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:05.849190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:05.849190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 650/1265] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:05.854010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:05.937844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:05.956221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf69385-b208-4c56-a5fe-7a230034e8de None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:05.989760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:05.990359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.990668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.990938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.991570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:05.991867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:05.993246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.004195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 616/1266] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:37:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:06.009926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.110852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d18b001-b402-460f-9261-6512f52a41e9 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.134350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:06.148469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.149090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.149364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.149629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.157837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.159283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:37:06.170965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:06.176464 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:06.183670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 651/1267] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:06.184390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.282528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e82846-fad4-42c6-b562-aa67ce307064 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.315867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.316468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.316745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.317014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.317731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:06.318032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.319596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.329881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:06.337821 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 617/1268] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:06.345060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.446425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea36dd7-1057-4435-9c72-13558c7febd9 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.480227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.480931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.481211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.481481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.491809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.493246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:37:06.507693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:06.513966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:37:06.515153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:37:06.515153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:06.515153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:06.515153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 652/1269] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:06.515638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.612526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5763c477-2f39-438b-a6a5-56ff37923ed2 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.646684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.647276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.647550 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.647814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.648438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:06.648734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.650221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.661093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 618/1270] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:06.666911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.698330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:06.766323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5b9fab-7a1b-47da-bd71-f46cd49e4e0d None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.798107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.798825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.799312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.799803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.807019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.811117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:37:06.820141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:06.821154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:06.822211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 653/1271] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:06.830043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.893994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:06.934137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a2de9-d517-43f1-a081-9cd75ae7c835 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.965847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61461555-8ed4-4fc9-81ea-d49d370be253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.966469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61461555-8ed4-4fc9-81ea-d49d370be253 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.966750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61461555-8ed4-4fc9-81ea-d49d370be253 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.967019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61461555-8ed4-4fc9-81ea-d49d370be253 None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.967697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61461555-8ed4-4fc9-81ea-d49d370be253 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:06.967998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61461555-8ed4-4fc9-81ea-d49d370be253 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.969414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61461555-8ed4-4fc9-81ea-d49d370be253 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.981110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 619/1272] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:06.987164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:07.087075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:07.095566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb9e17a-406d-4a5c-a7c5-cbba6732b379 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:07.129142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b322d58-8add-4eff-99a8-80cd453efb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:07.129848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b322d58-8add-4eff-99a8-80cd453efb73 None admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:07.130099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b322d58-8add-4eff-99a8-80cd453efb73 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:07.130372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b322d58-8add-4eff-99a8-80cd453efb73 None admin] PATH_INFO: `/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/659e82e28da54e28a58d9d6e9c5e4215/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:07.132247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b322d58-8add-4eff-99a8-80cd453efb73 None admin] RBAC: Authorizing `identity:create_grant(project_id=1b4ac0484ac7495f9bbc566d50e63fcf, user_id=659e82e28da54e28a58d9d6e9c5e4215, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:07.133666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b322d58-8add-4eff-99a8-80cd453efb73 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:07.222701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 654/1273] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Oct 4 08:37:06 2022] PUT /identity/v3/projects/1b4ac0484ac7495f9bbc566d50e63fcf/users/659e82e28da54e28a58d9d6e9c5e4215/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:09.382529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca3e47-a141-48d6-b230-9c12f81d236d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:09.383071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca3e47-a141-48d6-b230-9c12f81d236d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:09.383071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca3e47-a141-48d6-b230-9c12f81d236d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:09.384417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 620/1274] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:37:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:37:09.390979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c05b80-4dbd-4325-91f8-59cf47f10cd6 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:09.391434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c05b80-4dbd-4325-91f8-59cf47f10cd6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:09.391533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c05b80-4dbd-4325-91f8-59cf47f10cd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:09.411614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-66c05b80-4dbd-4325-91f8-59cf47f10cd6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:09.413956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-66c05b80-4dbd-4325-91f8-59cf47f10cd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:09.478483 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 655/1275] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:37:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:37:09.752616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca3e47-a141-48d6-b230-9c12f81d236d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:09.855668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca3e47-a141-48d6-b230-9c12f81d236d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:09.874149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:09.891791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:09.892531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:09.892784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:09.893058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:09.904306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:37:09.910782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:09.912006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:09.913062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 621/1276] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Oct 4 08:37:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:09.916279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c05b80-4dbd-4325-91f8-59cf47f10cd6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.000898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:10.019787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c05b80-4dbd-4325-91f8-59cf47f10cd6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:10.051716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:10.052314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.052587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.052853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.053501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] RBAC: Adding query filter params (name=admin) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:10.053794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:10.055185 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:10.064612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 656/1277] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Oct 4 08:37:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:10.070398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.169079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e130775-bca2-41d7-88be-0a9c05cbc062 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:10.202182 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:10.202861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.203140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.206843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.214102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:10.215984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:37:10.230757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:10.232077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:37:10.233272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:37:10.233272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:10.233272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:10.233272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 622/1278] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:10.237700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.352376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ceacc7-9b8f-4453-b662-9570e2b6f124 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:10.398295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:10.398985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.399248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.399516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.400173 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:10.400468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:10.401989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:10.415742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 657/1279] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:37:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:10.421728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.455907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:10.532804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6adc1b3-ee8b-4e9d-8530-ea73e743dc69 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:10.570627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:10.571272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.571545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.571812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] PATH_INFO: `/v3/users/placement` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.580606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:10.582058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:37:10.594486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:10.595675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:10.596726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 623/1280] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:10 2022] GET /identity/v3/users/placement => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:10.601522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.605595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:10.703013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144e47b0-ba3c-4df5-b91d-5e001494a107 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:10.738003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:10.738654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.738931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.739206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.739938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:10.740287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:10.741721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:10.761082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 658/1281] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Oct 4 08:37:10 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:10.768072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.858643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:10.871854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf28c3e-3fcd-4220-9c86-ca86c4958e8a None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:10.904106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:10.904705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.904983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.905252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.971541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:10.973006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.059359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:11.069689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 624/1282] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Oct 4 08:37:10 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:37:11.076024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.138196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:11.174888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84379c20-c05f-47c3-bd0e-d5bb52ef7bdd None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.206945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.207563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.207839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.208103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.218643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.220067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:37:11.235624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:11.236644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:37:11.237515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:37:11.237515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:37:11.237515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:11.237515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 659/1283] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:37:11.242669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.336503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f19691-b385-4a00-9ded-2b8d7219b347 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.370464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.371173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.371445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.371711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.372346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:11.372640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.374001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.385098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 625/1284] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:11.391154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.489498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882b90d4-c9ec-450a-9b91-62f1abdb3455 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.503120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:11.523430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.524060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.524336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.524607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] PATH_INFO: `/v3/projects/service` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.531583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.533021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:37:11.544049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:11.547403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: ERROR keystone.server.flask.application Oct 04 08:37:11.548520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 660/1285] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:11.551350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.600742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:11.647603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8702a6ac-a51d-4b88-b8e0-b223781c3c6f None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.680189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4088a93a-3c05-4fff-abfb-3f97749f00ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.680779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4088a93a-3c05-4fff-abfb-3f97749f00ba None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.681053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4088a93a-3c05-4fff-abfb-3f97749f00ba None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.681320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4088a93a-3c05-4fff-abfb-3f97749f00ba None admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.681991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4088a93a-3c05-4fff-abfb-3f97749f00ba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:11.682283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4088a93a-3c05-4fff-abfb-3f97749f00ba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.683942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4088a93a-3c05-4fff-abfb-3f97749f00ba None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.695426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 626/1286] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:11.702259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.801291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78429a49-883d-460d-9ca9-36daaf459189 None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.834969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce48516-9999-47a6-93f7-988c4417dcb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.835570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce48516-9999-47a6-93f7-988c4417dcb5 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.835845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce48516-9999-47a6-93f7-988c4417dcb5 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.836114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce48516-9999-47a6-93f7-988c4417dcb5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.836827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce48516-9999-47a6-93f7-988c4417dcb5 None admin] RBAC: Adding query filter params (role.id=1922dc727b87489999cc8916eacc229e, scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf, user.id=659e82e28da54e28a58d9d6e9c5e4215) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:11.837128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce48516-9999-47a6-93f7-988c4417dcb5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.838577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce48516-9999-47a6-93f7-988c4417dcb5 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.845010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 661/1287] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/role_assignments?user.id=659e82e28da54e28a58d9d6e9c5e4215&scope.project.id=1b4ac0484ac7495f9bbc566d50e63fcf&role.id=1922dc727b87489999cc8916eacc229e => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:14.042039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547e559-441f-4494-a56b-84ab133d08e6 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.043136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547e559-441f-4494-a56b-84ab133d08e6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.043230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7547e559-441f-4494-a56b-84ab133d08e6 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.044632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 627/1288] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:37:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:37:14.049906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab371995-53b4-48dc-b513-b5671333f383 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.050325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab371995-53b4-48dc-b513-b5671333f383 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.056111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab371995-53b4-48dc-b513-b5671333f383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.069200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ab371995-53b4-48dc-b513-b5671333f383 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:14.070372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ab371995-53b4-48dc-b513-b5671333f383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:14.098557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:14.136355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 662/1289] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:37:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:37:14.410332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7547e559-441f-4494-a56b-84ab133d08e6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:14.512768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7547e559-441f-4494-a56b-84ab133d08e6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:14.523218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:14.545686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:14.546301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.546598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.546861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] PATH_INFO: `/v3/services/placement` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.547427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:14.548767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:14.557387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:14.569218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:14.570291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 628/1290] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:37:14 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:14.571379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab371995-53b4-48dc-b513-b5671333f383 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:14.660285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab371995-53b4-48dc-b513-b5671333f383 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:14.692419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a021066-c111-4c99-a655-c9ae2dda0400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:14.693012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a021066-c111-4c99-a655-c9ae2dda0400 None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.693303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a021066-c111-4c99-a655-c9ae2dda0400 None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.693551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a021066-c111-4c99-a655-c9ae2dda0400 None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.694167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a021066-c111-4c99-a655-c9ae2dda0400 None admin] RBAC: Adding query filter params (name=placement) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:14.694487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a021066-c111-4c99-a655-c9ae2dda0400 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:14.695848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a021066-c111-4c99-a655-c9ae2dda0400 None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:14.701060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 663/1291] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Oct 4 08:37:14 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:14.707112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:14.804534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3aedca-b445-404a-947f-a7fa1a76bb1b None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:14.836902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31dae8-3d67-4df4-a650-f743dde031be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:14.837488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31dae8-3d67-4df4-a650-f743dde031be None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.837759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31dae8-3d67-4df4-a650-f743dde031be None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.838069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31dae8-3d67-4df4-a650-f743dde031be None admin] PATH_INFO: `/v3/services` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.838713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31dae8-3d67-4df4-a650-f743dde031be None admin] RBAC: Adding query filter params (type=placement) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:14.839011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31dae8-3d67-4df4-a650-f743dde031be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:14.841179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31dae8-3d67-4df4-a650-f743dde031be None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:14.845592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 629/1292] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Oct 4 08:37:14 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:17.003739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a285ecb-ff6c-4221-a15b-639710588658 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.004265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a285ecb-ff6c-4221-a15b-639710588658 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.004265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a285ecb-ff6c-4221-a15b-639710588658 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.005588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 664/1293] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:37:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:37:17.010755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baa72f8-d487-4c55-818f-3e8314540b3c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.011153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baa72f8-d487-4c55-818f-3e8314540b3c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.011331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baa72f8-d487-4c55-818f-3e8314540b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.029293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7baa72f8-d487-4c55-818f-3e8314540b3c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:17.030510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7baa72f8-d487-4c55-818f-3e8314540b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:17.060912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:17.097858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 630/1294] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:37:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:37:17.371886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a285ecb-ff6c-4221-a15b-639710588658 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:17.402370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:17.471458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a285ecb-ff6c-4221-a15b-639710588658 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:17.506488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8bd7b6-5ccd-46ad-8531-50d3f342a6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:17.507093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8bd7b6-5ccd-46ad-8531-50d3f342a6cb None admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.507368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8bd7b6-5ccd-46ad-8531-50d3f342a6cb None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.507637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8bd7b6-5ccd-46ad-8531-50d3f342a6cb None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.508309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8bd7b6-5ccd-46ad-8531-50d3f342a6cb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:17.509645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8bd7b6-5ccd-46ad-8531-50d3f342a6cb None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:17.594392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 665/1295] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Oct 4 08:37:17 2022] POST /identity/v3/services => generated 246 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:19.770187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9face5-cc00-4870-b958-fd60e52af654 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:19.770758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9face5-cc00-4870-b958-fd60e52af654 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:19.770758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9face5-cc00-4870-b958-fd60e52af654 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:19.772084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 631/1296] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:37:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:37:19.778005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe585d5-db89-4d57-ac78-0cb5ac1a6026 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:19.780426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe585d5-db89-4d57-ac78-0cb5ac1a6026 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:19.780426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe585d5-db89-4d57-ac78-0cb5ac1a6026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:19.797406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-bbe585d5-db89-4d57-ac78-0cb5ac1a6026 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:19.798661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-bbe585d5-db89-4d57-ac78-0cb5ac1a6026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:19.869615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 666/1297] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:37:19 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:37:20.135832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9face5-cc00-4870-b958-fd60e52af654 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:20.234108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9face5-cc00-4870-b958-fd60e52af654 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:20.267636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:20.268250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:20.268525 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:20.268807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:20.269486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:20.270875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:20.279575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:20.280914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:20.281884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 632/1298] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:37:20 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:20.286682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe585d5-db89-4d57-ac78-0cb5ac1a6026 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:20.304376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:20.383763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe585d5-db89-4d57-ac78-0cb5ac1a6026 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:20.418508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:20.419123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:20.421337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:20.421337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:20.421337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] RBAC: Adding query filter params (name=placement) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:20.421337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:20.421906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:20.426963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 667/1299] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Oct 4 08:37:20 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:20.433622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:20.464839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:20.531488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9433f-da6c-485a-81a6-d938c24d23d4 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:20.565665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c49221b-7584-4fa8-a439-e5af7928c90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:20.566257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c49221b-7584-4fa8-a439-e5af7928c90c None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:20.566563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c49221b-7584-4fa8-a439-e5af7928c90c None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:20.566833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c49221b-7584-4fa8-a439-e5af7928c90c None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:20.567523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c49221b-7584-4fa8-a439-e5af7928c90c None admin] RBAC: Adding query filter params (interface=public, service_id=e4d9f2e6f3344476a8e73287c8948c66, region_id=RegionOne) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:20.567827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c49221b-7584-4fa8-a439-e5af7928c90c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:20.569185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c49221b-7584-4fa8-a439-e5af7928c90c None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:20.574301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 633/1300] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Oct 4 08:37:20 2022] GET /identity/v3/endpoints?service_id=e4d9f2e6f3344476a8e73287c8948c66&interface=public®ion_id=RegionOne => generated 194 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:20.580016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:20.659416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:20.676740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43192f1f-73da-4d73-b3e4-eb3df6d3008e None admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:20.710177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dc5280-470d-4240-bf5c-2d44ca4a80bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:20.710798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc5280-470d-4240-bf5c-2d44ca4a80bf None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:20.711070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc5280-470d-4240-bf5c-2d44ca4a80bf None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:20.711332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc5280-470d-4240-bf5c-2d44ca4a80bf None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:20.711918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dc5280-470d-4240-bf5c-2d44ca4a80bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:20.713269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dc5280-470d-4240-bf5c-2d44ca4a80bf None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:20.719903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 668/1301] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Oct 4 08:37:20 2022] GET /identity/v3/services => generated 2191 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:22.872868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57177bfa-5c8d-4d89-8856-3685e5274777 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:22.873400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57177bfa-5c8d-4d89-8856-3685e5274777 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:22.873400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57177bfa-5c8d-4d89-8856-3685e5274777 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:22.874768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 634/1302] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:37:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:37:22.880004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f484e4e-e160-494c-9a96-89bd8e812e3b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:22.880411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f484e4e-e160-494c-9a96-89bd8e812e3b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:22.880546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f484e4e-e160-494c-9a96-89bd8e812e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:22.899497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1f484e4e-e160-494c-9a96-89bd8e812e3b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:22.900726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1f484e4e-e160-494c-9a96-89bd8e812e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:22.976469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 669/1303] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:37:22 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:37:23.257311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57177bfa-5c8d-4d89-8856-3685e5274777 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:23.351605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:23.358207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57177bfa-5c8d-4d89-8856-3685e5274777 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:23.391924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:23.392731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:23.392868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:23.393143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] PATH_INFO: `/v3/services/placement` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:23.393721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:23.395156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:37:23.403938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:37:23.405092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application with Lock( Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: ERROR keystone.server.flask.application Oct 04 08:37:23.406128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 635/1304] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Oct 4 08:37:23 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:37:23.411206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f484e4e-e160-494c-9a96-89bd8e812e3b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:23.485125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:23.507684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f484e4e-e160-494c-9a96-89bd8e812e3b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:23.539789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662810f2-2bf3-441a-8fac-3e536a99c1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:23.540388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662810f2-2bf3-441a-8fac-3e536a99c1db None admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:23.540661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662810f2-2bf3-441a-8fac-3e536a99c1db None admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:23.540929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662810f2-2bf3-441a-8fac-3e536a99c1db None admin] PATH_INFO: `/v3/services` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:23.541550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662810f2-2bf3-441a-8fac-3e536a99c1db None admin] RBAC: Adding query filter params (name=placement) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:23.542627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662810f2-2bf3-441a-8fac-3e536a99c1db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:23.545957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662810f2-2bf3-441a-8fac-3e536a99c1db None admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:23.549784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 670/1305] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Oct 4 08:37:23 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:23.556150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:23.653360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7319d0-35d8-45a6-bf98-9ed8e4c44144 None admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:23.686767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5508745-ef9f-4980-b763-5ba7b24450b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:23.687364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5508745-ef9f-4980-b763-5ba7b24450b3 None admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:23.687642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5508745-ef9f-4980-b763-5ba7b24450b3 None admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:23.687910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5508745-ef9f-4980-b763-5ba7b24450b3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:23.688563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5508745-ef9f-4980-b763-5ba7b24450b3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:23.689925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5508745-ef9f-4980-b763-5ba7b24450b3 None admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:23.821068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 636/1306] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:23 2022] POST /identity/v3/endpoints => generated 331 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:33.546899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=30 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:33.816434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:38.139405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05de4189-f7a2-4f23-a7a5-425bc82d6774 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:38.139984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05de4189-f7a2-4f23-a7a5-425bc82d6774 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:38.139984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05de4189-f7a2-4f23-a7a5-425bc82d6774 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:38.141270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 671/1307] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:43:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:38.147127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3bfe54-ebd3-4888-9677-ba9ff9c27318 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:38.148366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3bfe54-ebd3-4888-9677-ba9ff9c27318 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:38.148647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3bfe54-ebd3-4888-9677-ba9ff9c27318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:38.259966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-cc3bfe54-ebd3-4888-9677-ba9ff9c27318 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:38.261188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cc3bfe54-ebd3-4888-9677-ba9ff9c27318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:38.391261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 637/1308] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:38.699166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f988fa74-db74-46f7-a54c-bb6eeeb86698 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:38.699446 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f988fa74-db74-46f7-a54c-bb6eeeb86698 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:38.700484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f988fa74-db74-46f7-a54c-bb6eeeb86698 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:38.702323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 672/1309] 10.222.0.27 () {54 vars in 964 bytes} [Tue Oct 4 08:43:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:38.708466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9147392b-b2ab-4486-ba0a-89e1baa38d74 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:38.708829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9147392b-b2ab-4486-ba0a-89e1baa38d74 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:38.709174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9147392b-b2ab-4486-ba0a-89e1baa38d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:38.795805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9147392b-b2ab-4486-ba0a-89e1baa38d74 None None] MFA Rules not processed for user `df391d2bf7884538823f04db5564d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:38.797188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9147392b-b2ab-4486-ba0a-89e1baa38d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:38.819445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:38.880474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 638/1310] 10.222.0.27 () {58 vars in 1061 bytes} [Tue Oct 4 08:43:38 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:38.890287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f988fa74-db74-46f7-a54c-bb6eeeb86698 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:38.892204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f988fa74-db74-46f7-a54c-bb6eeeb86698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:39.011690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f988fa74-db74-46f7-a54c-bb6eeeb86698 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:43:39.091028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dfa0df-88bd-49db-b17e-db6575136cbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:43:39.091660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dfa0df-88bd-49db-b17e-db6575136cbf service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:39.091938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dfa0df-88bd-49db-b17e-db6575136cbf service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:39.092214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dfa0df-88bd-49db-b17e-db6575136cbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:39.093820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-19dfa0df-88bd-49db-b17e-db6575136cbf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:39.152159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:39.211235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dfa0df-88bd-49db-b17e-db6575136cbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:43:39.212675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dfa0df-88bd-49db-b17e-db6575136cbf service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:43:39.232655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 673/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:43:41.713458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e675cd43-1956-48a9-9bfd-c50adb3159e0 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:41.714045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e675cd43-1956-48a9-9bfd-c50adb3159e0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:41.714141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e675cd43-1956-48a9-9bfd-c50adb3159e0 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:41.715535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 639/1312] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:43:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:41.720788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f97ee-9305-42f9-8510-129965c10502 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:41.721214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f97ee-9305-42f9-8510-129965c10502 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:41.721334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f97ee-9305-42f9-8510-129965c10502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:41.790765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-022f97ee-9305-42f9-8510-129965c10502 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:41.791964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-022f97ee-9305-42f9-8510-129965c10502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:41.888068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 674/1313] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:43:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:42.169736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e675cd43-1956-48a9-9bfd-c50adb3159e0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:42.171317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e675cd43-1956-48a9-9bfd-c50adb3159e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:42.276785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e675cd43-1956-48a9-9bfd-c50adb3159e0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:43:42.351100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:42.353417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb3eee6-aa2d-4c98-bb66-4993524c8427 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:43:42.354021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb3eee6-aa2d-4c98-bb66-4993524c8427 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:42.354298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb3eee6-aa2d-4c98-bb66-4993524c8427 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:42.354591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb3eee6-aa2d-4c98-bb66-4993524c8427 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:42.356925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8cb3eee6-aa2d-4c98-bb66-4993524c8427 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:42.467200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb3eee6-aa2d-4c98-bb66-4993524c8427 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:43:42.468555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb3eee6-aa2d-4c98-bb66-4993524c8427 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:43:42.486848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 640/1314] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:43:45.196433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ed18b5-16a0-4781-96b4-1e637bf22375 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:45.196999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ed18b5-16a0-4781-96b4-1e637bf22375 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:45.196999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ed18b5-16a0-4781-96b4-1e637bf22375 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:45.198284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 675/1315] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:43:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:45.204089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d462471e-c9e9-4a13-87ab-7214e0263ce2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:45.204500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d462471e-c9e9-4a13-87ab-7214e0263ce2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:45.204618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d462471e-c9e9-4a13-87ab-7214e0263ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:45.269684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d462471e-c9e9-4a13-87ab-7214e0263ce2 None None] MFA Rules not processed for user `547771dc36184db8b769aa9879b27b74`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:45.270912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d462471e-c9e9-4a13-87ab-7214e0263ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:45.320922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:45.364762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 641/1316] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:43:45 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:45.664299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee606ca-9baf-4cc5-bce6-4ae5cfd0da5e None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:45.664608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee606ca-9baf-4cc5-bce6-4ae5cfd0da5e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:45.664879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee606ca-9baf-4cc5-bce6-4ae5cfd0da5e None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:45.666194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 676/1317] 10.222.0.27 () {54 vars in 964 bytes} [Tue Oct 4 08:43:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:45.672038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe060790-3d5f-46c1-aa6d-26d4480cea14 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:45.672326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe060790-3d5f-46c1-aa6d-26d4480cea14 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:45.672603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe060790-3d5f-46c1-aa6d-26d4480cea14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:45.746047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-fe060790-3d5f-46c1-aa6d-26d4480cea14 None None] MFA Rules not processed for user `df391d2bf7884538823f04db5564d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:45.747383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fe060790-3d5f-46c1-aa6d-26d4480cea14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:45.820966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 642/1318] 10.222.0.27 () {58 vars in 1061 bytes} [Tue Oct 4 08:43:45 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:45.830532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee606ca-9baf-4cc5-bce6-4ae5cfd0da5e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:45.832043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fee606ca-9baf-4cc5-bce6-4ae5cfd0da5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:45.842250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:45.931966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee606ca-9baf-4cc5-bce6-4ae5cfd0da5e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:43:46.005488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964da878-109d-421f-948e-310b2df89d30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:43:46.006086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964da878-109d-421f-948e-310b2df89d30 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:46.006362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964da878-109d-421f-948e-310b2df89d30 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:46.006657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964da878-109d-421f-948e-310b2df89d30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:46.008229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-964da878-109d-421f-948e-310b2df89d30 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:46.068328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:46.113246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964da878-109d-421f-948e-310b2df89d30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:43:46.114704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964da878-109d-421f-948e-310b2df89d30 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:43:46.133168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 677/1319] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:45 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:43:49.486367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a48afb-0a50-4c1e-be1c-eb8fdcbd4d3f None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:49.486917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a48afb-0a50-4c1e-be1c-eb8fdcbd4d3f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:49.486917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a48afb-0a50-4c1e-be1c-eb8fdcbd4d3f None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:49.488268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 643/1320] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:43:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:49.493547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b13853d-4a15-433c-8b54-c75822d4dee3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:49.493938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b13853d-4a15-433c-8b54-c75822d4dee3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:49.494091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b13853d-4a15-433c-8b54-c75822d4dee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:49.559873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6b13853d-4a15-433c-8b54-c75822d4dee3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:49.561078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6b13853d-4a15-433c-8b54-c75822d4dee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:49.655860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 678/1321] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:43:49 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:49.939633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a48afb-0a50-4c1e-be1c-eb8fdcbd4d3f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:49.941220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-83a48afb-0a50-4c1e-be1c-eb8fdcbd4d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:49.992405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:50.045096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a48afb-0a50-4c1e-be1c-eb8fdcbd4d3f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:43:50.118274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fac395-1f94-4862-b76b-04e24e0710dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:43:50.119027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fac395-1f94-4862-b76b-04e24e0710dd service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:50.119303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fac395-1f94-4862-b76b-04e24e0710dd service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:50.119569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fac395-1f94-4862-b76b-04e24e0710dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:50.121214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-15fac395-1f94-4862-b76b-04e24e0710dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:50.213746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:50.226427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fac395-1f94-4862-b76b-04e24e0710dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:43:50.227839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fac395-1f94-4862-b76b-04e24e0710dd service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:43:50.247351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 644/1322] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:49 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:43:53.416306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8bc162-69fe-4203-9c73-46c2b6692000 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:53.416821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8bc162-69fe-4203-9c73-46c2b6692000 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:53.416916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8bc162-69fe-4203-9c73-46c2b6692000 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:53.418247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 679/1323] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:43:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:53.423700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f2e31-14a8-4720-b872-a53f40d71ed0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:53.424117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f2e31-14a8-4720-b872-a53f40d71ed0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:53.424316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f2e31-14a8-4720-b872-a53f40d71ed0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:53.490829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6b5f2e31-14a8-4720-b872-a53f40d71ed0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:53.491969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6b5f2e31-14a8-4720-b872-a53f40d71ed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:53.596652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 645/1324] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:43:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:53.884238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8bc162-69fe-4203-9c73-46c2b6692000 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:53.885773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fa8bc162-69fe-4203-9c73-46c2b6692000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:53.906097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:53.989454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8bc162-69fe-4203-9c73-46c2b6692000 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:43:54.068691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd71d40-c56c-4ec7-a899-652982e95df2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:43:54.069698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd71d40-c56c-4ec7-a899-652982e95df2 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:54.070026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd71d40-c56c-4ec7-a899-652982e95df2 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:54.070375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd71d40-c56c-4ec7-a899-652982e95df2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:54.072036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-dcd71d40-c56c-4ec7-a899-652982e95df2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:54.161011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:54.197895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd71d40-c56c-4ec7-a899-652982e95df2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:43:54.199503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd71d40-c56c-4ec7-a899-652982e95df2 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:43:54.218139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 680/1325] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:43:56.448494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cdfde2-36ad-425e-9878-c7752871e19d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:56.449031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cdfde2-36ad-425e-9878-c7752871e19d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:56.449031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cdfde2-36ad-425e-9878-c7752871e19d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:56.450979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 646/1326] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:43:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:56.456536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d39a29-fe6e-4800-ae9f-f07de0236dc0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:56.457006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d39a29-fe6e-4800-ae9f-f07de0236dc0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:56.457212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d39a29-fe6e-4800-ae9f-f07de0236dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:56.527732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-47d39a29-fe6e-4800-ae9f-f07de0236dc0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:56.529323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-47d39a29-fe6e-4800-ae9f-f07de0236dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:56.627058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 681/1327] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:43:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:56.913032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cdfde2-36ad-425e-9878-c7752871e19d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:56.914608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b1cdfde2-36ad-425e-9878-c7752871e19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:56.975755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:57.019332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cdfde2-36ad-425e-9878-c7752871e19d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:43:57.092893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebb5f18-15bb-464a-827e-a3909818af0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:43:57.093584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb5f18-15bb-464a-827e-a3909818af0b service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:57.093863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb5f18-15bb-464a-827e-a3909818af0b service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:57.094134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb5f18-15bb-464a-827e-a3909818af0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:57.095766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2ebb5f18-15bb-464a-827e-a3909818af0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:57.204017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:57.206374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebb5f18-15bb-464a-827e-a3909818af0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:43:57.207836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebb5f18-15bb-464a-827e-a3909818af0b service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:43:57.226223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 647/1328] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:43:59.464540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37645b36-cb8b-4e17-9637-9a115b42a9bf None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:59.465601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37645b36-cb8b-4e17-9637-9a115b42a9bf None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:59.465601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37645b36-cb8b-4e17-9637-9a115b42a9bf None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:59.467126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 682/1329] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:43:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:43:59.472397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747e70ee-860f-4ef3-a779-4ee657951d7b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:59.472843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747e70ee-860f-4ef3-a779-4ee657951d7b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:59.472945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747e70ee-860f-4ef3-a779-4ee657951d7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:59.538624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-747e70ee-860f-4ef3-a779-4ee657951d7b None None] MFA Rules not processed for user `547771dc36184db8b769aa9879b27b74`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:43:59.539844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-747e70ee-860f-4ef3-a779-4ee657951d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:59.636491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 648/1330] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:43:59 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:43:59.925090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37645b36-cb8b-4e17-9637-9a115b42a9bf None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:59.926690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-37645b36-cb8b-4e17-9637-9a115b42a9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:59.959046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:00.045520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37645b36-cb8b-4e17-9637-9a115b42a9bf None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:00.133458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a6ae7b-1074-4822-b19f-6ee0fcf1e722 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:00.134063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6ae7b-1074-4822-b19f-6ee0fcf1e722 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:00.134338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6ae7b-1074-4822-b19f-6ee0fcf1e722 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:00.134635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6ae7b-1074-4822-b19f-6ee0fcf1e722 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:00.136779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c8a6ae7b-1074-4822-b19f-6ee0fcf1e722 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:00.212979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:00.244646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a6ae7b-1074-4822-b19f-6ee0fcf1e722 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:00.246070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a6ae7b-1074-4822-b19f-6ee0fcf1e722 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:00.265204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 683/1331] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:59 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:02.854034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad2f97-7b65-4089-b166-c7e730ac51ab None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:02.854633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad2f97-7b65-4089-b166-c7e730ac51ab None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:02.854633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad2f97-7b65-4089-b166-c7e730ac51ab None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:02.856482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 649/1332] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:44:02.861641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506a0f58-0c29-4494-b0c7-2d5bf63e8a86 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:02.862029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506a0f58-0c29-4494-b0c7-2d5bf63e8a86 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:02.862212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506a0f58-0c29-4494-b0c7-2d5bf63e8a86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:02.927691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-506a0f58-0c29-4494-b0c7-2d5bf63e8a86 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:02.928895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-506a0f58-0c29-4494-b0c7-2d5bf63e8a86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:03.025626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 684/1333] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:03.312576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad2f97-7b65-4089-b166-c7e730ac51ab None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:03.314151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f0ad2f97-7b65-4089-b166-c7e730ac51ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:03.383828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:03.416884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad2f97-7b65-4089-b166-c7e730ac51ab None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:03.489941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef522b2-ab61-4927-906a-bdf574c0e32f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:03.490509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef522b2-ab61-4927-906a-bdf574c0e32f service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:03.490790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef522b2-ab61-4927-906a-bdf574c0e32f service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:03.491058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef522b2-ab61-4927-906a-bdf574c0e32f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:03.492639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4ef522b2-ab61-4927-906a-bdf574c0e32f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:03.600614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef522b2-ab61-4927-906a-bdf574c0e32f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:03.602086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef522b2-ab61-4927-906a-bdf574c0e32f service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:03.612629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:03.620825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 650/1334] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:05.810844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad624aa4-e3aa-4d5f-b85b-ef07616ec266 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:05.811378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad624aa4-e3aa-4d5f-b85b-ef07616ec266 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:05.811378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad624aa4-e3aa-4d5f-b85b-ef07616ec266 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:05.812713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 685/1335] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:44:05.817945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958c6c5-a336-4b3e-8407-1cfda4cc8fb3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:05.818347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958c6c5-a336-4b3e-8407-1cfda4cc8fb3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:05.818518 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958c6c5-a336-4b3e-8407-1cfda4cc8fb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:05.886294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b958c6c5-a336-4b3e-8407-1cfda4cc8fb3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:05.888583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b958c6c5-a336-4b3e-8407-1cfda4cc8fb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:05.982930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 651/1336] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:06.270209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad624aa4-e3aa-4d5f-b85b-ef07616ec266 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:06.271759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ad624aa4-e3aa-4d5f-b85b-ef07616ec266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:06.310981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:06.372896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad624aa4-e3aa-4d5f-b85b-ef07616ec266 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:06.445646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416cc72e-b285-4752-9561-16d73cbe0752 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:06.446245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416cc72e-b285-4752-9561-16d73cbe0752 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:06.446554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416cc72e-b285-4752-9561-16d73cbe0752 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:06.446828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416cc72e-b285-4752-9561-16d73cbe0752 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:06.448408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-416cc72e-b285-4752-9561-16d73cbe0752 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:06.537323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:06.562596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416cc72e-b285-4752-9561-16d73cbe0752 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:06.564026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416cc72e-b285-4752-9561-16d73cbe0752 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:06.582494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 686/1337] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:09.300290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd65853-300e-4d8b-93af-8056efaa0a16 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:09.300811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd65853-300e-4d8b-93af-8056efaa0a16 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:09.300811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd65853-300e-4d8b-93af-8056efaa0a16 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:09.302153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 652/1338] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:09.307369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777a9dd2-1ae5-4dd6-941d-2fd4a674b775 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:09.307769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777a9dd2-1ae5-4dd6-941d-2fd4a674b775 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:09.307908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777a9dd2-1ae5-4dd6-941d-2fd4a674b775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:09.374677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-777a9dd2-1ae5-4dd6-941d-2fd4a674b775 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:09.375896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-777a9dd2-1ae5-4dd6-941d-2fd4a674b775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:09.472078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 687/1339] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:09 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:09.764757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd65853-300e-4d8b-93af-8056efaa0a16 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:09.766315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7cd65853-300e-4d8b-93af-8056efaa0a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:09.842050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:09.869297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd65853-300e-4d8b-93af-8056efaa0a16 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:09.947294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c44d7e-23b0-4c2c-bd34-03648f98f35a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:09.947902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c44d7e-23b0-4c2c-bd34-03648f98f35a service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:09.948177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c44d7e-23b0-4c2c-bd34-03648f98f35a service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:09.948445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c44d7e-23b0-4c2c-bd34-03648f98f35a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:09.951014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-27c44d7e-23b0-4c2c-bd34-03648f98f35a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:10.059231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c44d7e-23b0-4c2c-bd34-03648f98f35a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:10.060688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c44d7e-23b0-4c2c-bd34-03648f98f35a service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:10.078705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 653/1340] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:14.657150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70727ea2-8e6e-4758-adc5-33f92c5e1e61 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:14.657738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70727ea2-8e6e-4758-adc5-33f92c5e1e61 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:14.657738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70727ea2-8e6e-4758-adc5-33f92c5e1e61 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:14.659113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 688/1341] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:14.664428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc6df0-8cb8-4661-85f6-6e8c33c748d3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:14.665209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc6df0-8cb8-4661-85f6-6e8c33c748d3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:14.665484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc6df0-8cb8-4661-85f6-6e8c33c748d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:14.671805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:14.734809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d5fc6df0-8cb8-4661-85f6-6e8c33c748d3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:14.736027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d5fc6df0-8cb8-4661-85f6-6e8c33c748d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:14.837188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 654/1342] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:15.130794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70727ea2-8e6e-4758-adc5-33f92c5e1e61 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:15.132394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-70727ea2-8e6e-4758-adc5-33f92c5e1e61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:15.189438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:15.251280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70727ea2-8e6e-4758-adc5-33f92c5e1e61 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:15.324909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69686460-bb40-4c81-96da-e6764dcf9678 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:15.325599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69686460-bb40-4c81-96da-e6764dcf9678 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:15.325873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69686460-bb40-4c81-96da-e6764dcf9678 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:15.326138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69686460-bb40-4c81-96da-e6764dcf9678 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:15.327784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-69686460-bb40-4c81-96da-e6764dcf9678 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:15.422888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:15.438717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69686460-bb40-4c81-96da-e6764dcf9678 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:15.440142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69686460-bb40-4c81-96da-e6764dcf9678 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:15.458236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 689/1343] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:18.583579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47c80d-7ec6-4a3d-a13f-02ad03bde6d1 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:18.584149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47c80d-7ec6-4a3d-a13f-02ad03bde6d1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:18.584149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47c80d-7ec6-4a3d-a13f-02ad03bde6d1 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:18.585458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 655/1344] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:18.590766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f1df8-022a-445a-aa33-345a07963753 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:18.591240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f1df8-022a-445a-aa33-345a07963753 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:18.591349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f1df8-022a-445a-aa33-345a07963753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:18.661156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-136f1df8-022a-445a-aa33-345a07963753 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:18.662299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-136f1df8-022a-445a-aa33-345a07963753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:18.767586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 690/1345] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:18 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:19.054071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d47c80d-7ec6-4a3d-a13f-02ad03bde6d1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:19.055624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5d47c80d-7ec6-4a3d-a13f-02ad03bde6d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:19.143616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:19.160766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d47c80d-7ec6-4a3d-a13f-02ad03bde6d1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:19.238354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e7b1f3-931f-402b-a208-847e3fc83363 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:19.238986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7b1f3-931f-402b-a208-847e3fc83363 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:19.239262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7b1f3-931f-402b-a208-847e3fc83363 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:19.239528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7b1f3-931f-402b-a208-847e3fc83363 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:19.241127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-19e7b1f3-931f-402b-a208-847e3fc83363 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:19.352536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e7b1f3-931f-402b-a208-847e3fc83363 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:19.353989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e7b1f3-931f-402b-a208-847e3fc83363 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:19.395504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 656/1346] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:24.205189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc46c943-44da-4f00-9e08-23c9940ef487 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:24.205724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc46c943-44da-4f00-9e08-23c9940ef487 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:24.205724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc46c943-44da-4f00-9e08-23c9940ef487 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:24.207104 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 691/1347] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:24.212387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062575dc-60d3-468d-8062-4c6a4c6b53f6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:24.212801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062575dc-60d3-468d-8062-4c6a4c6b53f6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:24.213151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062575dc-60d3-468d-8062-4c6a4c6b53f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:24.228983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:24.283841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-062575dc-60d3-468d-8062-4c6a4c6b53f6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:24.284995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-062575dc-60d3-468d-8062-4c6a4c6b53f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:24.390788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 657/1348] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:24.690352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc46c943-44da-4f00-9e08-23c9940ef487 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:24.691917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-dc46c943-44da-4f00-9e08-23c9940ef487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:24.750608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:24.803633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc46c943-44da-4f00-9e08-23c9940ef487 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:24.880189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04917459-5da7-4256-9f45-4bc637aeb789 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:24.880785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04917459-5da7-4256-9f45-4bc637aeb789 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:24.881058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04917459-5da7-4256-9f45-4bc637aeb789 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:24.881324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04917459-5da7-4256-9f45-4bc637aeb789 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:24.882930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-04917459-5da7-4256-9f45-4bc637aeb789 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:24.989435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:24.994059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04917459-5da7-4256-9f45-4bc637aeb789 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:24.996134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04917459-5da7-4256-9f45-4bc637aeb789 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:25.015722 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 692/1349] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:30.141612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c146775-a889-4d18-adbd-bf29577330ff None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:30.141612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c146775-a889-4d18-adbd-bf29577330ff None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:30.141612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c146775-a889-4d18-adbd-bf29577330ff None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:30.141612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 658/1350] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:44:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:30.148993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a06c0-9d41-4c02-acd3-a351d1bc310f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:30.149731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a06c0-9d41-4c02-acd3-a351d1bc310f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:30.150161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a06c0-9d41-4c02-acd3-a351d1bc310f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:30.299418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-446a06c0-9d41-4c02-acd3-a351d1bc310f None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:30.300620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-446a06c0-9d41-4c02-acd3-a351d1bc310f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:30.448915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 693/1351] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:44:30 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:34.390526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=59 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:34.536403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a522b9-2b4e-4310-83a2-b09e5826ea2f None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:34.536951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a522b9-2b4e-4310-83a2-b09e5826ea2f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:34.537380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a522b9-2b4e-4310-83a2-b09e5826ea2f None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:34.538928 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 659/1352] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:44:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:34.543841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04473d1f-caaf-4bcb-9492-f13fe53666c6 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:34.544320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04473d1f-caaf-4bcb-9492-f13fe53666c6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:34.544419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04473d1f-caaf-4bcb-9492-f13fe53666c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:34.618857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:34.644085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-04473d1f-caaf-4bcb-9492-f13fe53666c6 None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:34.645333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-04473d1f-caaf-4bcb-9492-f13fe53666c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:34.760990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 694/1353] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:44:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:36.447593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201ae10-bd46-4bd6-9cdb-375089cd98d6 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:36.448114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201ae10-bd46-4bd6-9cdb-375089cd98d6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:36.448257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201ae10-bd46-4bd6-9cdb-375089cd98d6 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:36.450164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 660/1354] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:44:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:36.455823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e36740-0843-4e56-889d-137799bce1c1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:36.456284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e36740-0843-4e56-889d-137799bce1c1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:36.456398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e36740-0843-4e56-889d-137799bce1c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:36.535269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-69e36740-0843-4e56-889d-137799bce1c1 None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:36.536492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-69e36740-0843-4e56-889d-137799bce1c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:36.579299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:36.645373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 695/1355] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:44:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:40.192613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c7429d-0178-4eb5-ba78-ab3eab319799 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:40.193123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c7429d-0178-4eb5-ba78-ab3eab319799 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:40.193311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c7429d-0178-4eb5-ba78-ab3eab319799 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:40.194919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 661/1356] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:40.201094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4a409-5825-4d75-9287-744058b41051 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:40.201592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4a409-5825-4d75-9287-744058b41051 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:40.201707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b4a409-5825-4d75-9287-744058b41051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:40.279869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-61b4a409-5825-4d75-9287-744058b41051 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:40.281090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-61b4a409-5825-4d75-9287-744058b41051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:40.386465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 696/1357] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:40.700789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0505092-a507-4305-999c-e656dc8eca14 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:40.701199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0505092-a507-4305-999c-e656dc8eca14 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:40.701752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0505092-a507-4305-999c-e656dc8eca14 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:40.703829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 662/1358] 10.222.0.27 () {54 vars in 961 bytes} [Tue Oct 4 08:44:40 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:40.712390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf3c7c-9a5f-4220-b42f-e48f702ebcca None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:40.712807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf3c7c-9a5f-4220-b42f-e48f702ebcca None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:40.713183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf3c7c-9a5f-4220-b42f-e48f702ebcca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:40.734271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:40.798122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0abf3c7c-9a5f-4220-b42f-e48f702ebcca None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:40.799560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0abf3c7c-9a5f-4220-b42f-e48f702ebcca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:40.932610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 697/1359] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Oct 4 08:44:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:44:40.941410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0505092-a507-4305-999c-e656dc8eca14 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:40.942973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f0505092-a507-4305-999c-e656dc8eca14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:41.118831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0505092-a507-4305-999c-e656dc8eca14 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:41.195006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:41.236756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f854f3-02c7-4c7a-9ebd-cc9cc05105e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:41.237496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f854f3-02c7-4c7a-9ebd-cc9cc05105e4 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:41.237869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f854f3-02c7-4c7a-9ebd-cc9cc05105e4 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:41.238233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f854f3-02c7-4c7a-9ebd-cc9cc05105e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:41.239938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-65f854f3-02c7-4c7a-9ebd-cc9cc05105e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:41.335134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f854f3-02c7-4c7a-9ebd-cc9cc05105e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:41.336769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f854f3-02c7-4c7a-9ebd-cc9cc05105e4 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:41.360700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 663/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:44:43.818554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd8261-0d19-47e3-ab08-bfd67e9dbab4 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:43.819099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd8261-0d19-47e3-ab08-bfd67e9dbab4 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:43.819195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd8261-0d19-47e3-ab08-bfd67e9dbab4 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:43.820970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 698/1361] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:43.827241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee2cfb-a3a2-4a86-b227-7ddbce6cffbd None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:43.827712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee2cfb-a3a2-4a86-b227-7ddbce6cffbd None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:43.827712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee2cfb-a3a2-4a86-b227-7ddbce6cffbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:43.897803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d7ee2cfb-a3a2-4a86-b227-7ddbce6cffbd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:43.898311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d7ee2cfb-a3a2-4a86-b227-7ddbce6cffbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:43.938258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:44.004033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 664/1362] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:44.300088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fd8261-0d19-47e3-ab08-bfd67e9dbab4 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:44.301696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f0fd8261-0d19-47e3-ab08-bfd67e9dbab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:44.381909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:44.480870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fd8261-0d19-47e3-ab08-bfd67e9dbab4 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:44.615012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89582c98-507b-4a58-a176-46290aafdd73 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:44.615012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89582c98-507b-4a58-a176-46290aafdd73 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:44.615012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89582c98-507b-4a58-a176-46290aafdd73 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:44.618709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89582c98-507b-4a58-a176-46290aafdd73 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:44.618709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-89582c98-507b-4a58-a176-46290aafdd73 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:44.645239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:44.721031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89582c98-507b-4a58-a176-46290aafdd73 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:44.722638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89582c98-507b-4a58-a176-46290aafdd73 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:44.742002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 699/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:47.399720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453b31d-4313-46f8-b9d1-610660b88236 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:47.400183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453b31d-4313-46f8-b9d1-610660b88236 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:47.400277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453b31d-4313-46f8-b9d1-610660b88236 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:47.401664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 665/1364] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:47.407055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b62cc0c-55cf-492f-b881-f442d3231c5d None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:47.407503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b62cc0c-55cf-492f-b881-f442d3231c5d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:47.407594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b62cc0c-55cf-492f-b881-f442d3231c5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:47.477656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0b62cc0c-55cf-492f-b881-f442d3231c5d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:47.479050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0b62cc0c-55cf-492f-b881-f442d3231c5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:47.582107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:47.584585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 700/1365] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:47.881462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1af09a-87e4-4491-ab61-8994e651266e None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:47.882089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1af09a-87e4-4491-ab61-8994e651266e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:47.882650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1af09a-87e4-4491-ab61-8994e651266e None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:47.884214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 666/1366] 10.222.0.27 () {54 vars in 961 bytes} [Tue Oct 4 08:44:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:47.890555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66889433-fefa-4fc0-ae20-bd23561a09b2 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:47.890842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66889433-fefa-4fc0-ae20-bd23561a09b2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:47.891169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66889433-fefa-4fc0-ae20-bd23561a09b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:47.982687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-66889433-fefa-4fc0-ae20-bd23561a09b2 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:47.984090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-66889433-fefa-4fc0-ae20-bd23561a09b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:48.101051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 701/1367] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Oct 4 08:44:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:44:48.109306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1af09a-87e4-4491-ab61-8994e651266e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:48.110859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-db1af09a-87e4-4491-ab61-8994e651266e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:48.269759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1af09a-87e4-4491-ab61-8994e651266e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:48.290617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:48.395814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3571016-877c-4040-9046-9e7b3cd083cb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:48.396429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3571016-877c-4040-9046-9e7b3cd083cb service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:48.396705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3571016-877c-4040-9046-9e7b3cd083cb service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:48.396971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3571016-877c-4040-9046-9e7b3cd083cb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:48.398610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f3571016-877c-4040-9046-9e7b3cd083cb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:48.498556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3571016-877c-4040-9046-9e7b3cd083cb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:48.500149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3571016-877c-4040-9046-9e7b3cd083cb service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:48.519616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 667/1368] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:48.585395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77e71b1-4fc4-4588-a52c-0acbb0f451e6 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:48.585927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77e71b1-4fc4-4588-a52c-0acbb0f451e6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:48.585927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77e71b1-4fc4-4588-a52c-0acbb0f451e6 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:48.587366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 702/1369] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:44:48.593730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d4a6f-e670-4297-b32b-8cfcef272418 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:48.594044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d4a6f-e670-4297-b32b-8cfcef272418 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:48.594337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d4a6f-e670-4297-b32b-8cfcef272418 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:48.634146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:48.670496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a78d4a6f-e670-4297-b32b-8cfcef272418 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:48.671949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a78d4a6f-e670-4297-b32b-8cfcef272418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:48.776145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 668/1370] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:49.083310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec5fa4-3ad4-4ea1-8ee4-b83ae728dd36 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:49.083816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec5fa4-3ad4-4ea1-8ee4-b83ae728dd36 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:49.084364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fec5fa4-3ad4-4ea1-8ee4-b83ae728dd36 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:49.086125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 703/1371] 10.222.0.27 () {54 vars in 963 bytes} [Tue Oct 4 08:44:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:49.091056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b014c9-51dc-4b47-bd64-0e1b2bfe9b41 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:49.091414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b014c9-51dc-4b47-bd64-0e1b2bfe9b41 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:49.091761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b014c9-51dc-4b47-bd64-0e1b2bfe9b41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:49.181679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-55b014c9-51dc-4b47-bd64-0e1b2bfe9b41 None None] MFA Rules not processed for user `f49a19f9484d4aa99567e66b4fc396ef`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:49.183687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-55b014c9-51dc-4b47-bd64-0e1b2bfe9b41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:49.242111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:49.275251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 669/1372] 10.222.0.27 () {58 vars in 1060 bytes} [Tue Oct 4 08:44:49 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:49.283685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fec5fa4-3ad4-4ea1-8ee4-b83ae728dd36 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:49.285179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8fec5fa4-3ad4-4ea1-8ee4-b83ae728dd36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:49.338710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:49.406580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fec5fa4-3ad4-4ea1-8ee4-b83ae728dd36 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:49.484104 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3418339a-9aae-424c-9a64-38e20a4901bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:49.484715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418339a-9aae-424c-9a64-38e20a4901bc service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:49.484988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418339a-9aae-424c-9a64-38e20a4901bc service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:49.485254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418339a-9aae-424c-9a64-38e20a4901bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:49.486885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3418339a-9aae-424c-9a64-38e20a4901bc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:49.598861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:49.615748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3418339a-9aae-424c-9a64-38e20a4901bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:49.617366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3418339a-9aae-424c-9a64-38e20a4901bc service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:49.638825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 704/1373] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:50.959044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05057b02-9c8e-4235-bfbe-a329a102784d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:50.959620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05057b02-9c8e-4235-bfbe-a329a102784d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:50.959620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05057b02-9c8e-4235-bfbe-a329a102784d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:50.961061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 670/1374] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:50.967087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1592ef36-1696-4551-b20b-f221e55eb89f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:50.967480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1592ef36-1696-4551-b20b-f221e55eb89f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:50.967665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1592ef36-1696-4551-b20b-f221e55eb89f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:51.032675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1592ef36-1696-4551-b20b-f221e55eb89f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:51.033885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1592ef36-1696-4551-b20b-f221e55eb89f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:51.134043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 705/1375] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:51.439532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05057b02-9c8e-4235-bfbe-a329a102784d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:51.441133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-05057b02-9c8e-4235-bfbe-a329a102784d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:51.598457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05057b02-9c8e-4235-bfbe-a329a102784d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:51.665201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:51.753536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e9b6a-ef1f-4ce4-b10b-51df3c30c1a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:51.754179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e9b6a-ef1f-4ce4-b10b-51df3c30c1a9 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:51.754583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e9b6a-ef1f-4ce4-b10b-51df3c30c1a9 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:51.754855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e9b6a-ef1f-4ce4-b10b-51df3c30c1a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:51.756457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-077e9b6a-ef1f-4ce4-b10b-51df3c30c1a9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:51.859751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e9b6a-ef1f-4ce4-b10b-51df3c30c1a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:51.861328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e9b6a-ef1f-4ce4-b10b-51df3c30c1a9 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:51.884416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 671/1376] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:44:54.271935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc74f1-8308-4305-b3bd-190a8ae76e5e None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:54.272491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc74f1-8308-4305-b3bd-190a8ae76e5e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:54.272491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc74f1-8308-4305-b3bd-190a8ae76e5e None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:54.273793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 706/1377] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:54.279305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc39001-86fc-4827-92a7-6ac16b5c996f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:54.279722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc39001-86fc-4827-92a7-6ac16b5c996f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:54.279822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc39001-86fc-4827-92a7-6ac16b5c996f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:54.346784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-afc39001-86fc-4827-92a7-6ac16b5c996f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:54.347981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-afc39001-86fc-4827-92a7-6ac16b5c996f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:54.355156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:54.453258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 672/1378] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:44:54.743422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc74f1-8308-4305-b3bd-190a8ae76e5e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:54.746211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f9fc74f1-8308-4305-b3bd-190a8ae76e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:54.811751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:54.902575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc74f1-8308-4305-b3bd-190a8ae76e5e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:55.023016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aa9f78-fcc6-4f47-bf5b-a867d3d6498f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:55.023544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aa9f78-fcc6-4f47-bf5b-a867d3d6498f service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:55.023825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aa9f78-fcc6-4f47-bf5b-a867d3d6498f service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:55.024093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aa9f78-fcc6-4f47-bf5b-a867d3d6498f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:55.025711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-96aa9f78-fcc6-4f47-bf5b-a867d3d6498f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:55.046816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:55.123520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aa9f78-fcc6-4f47-bf5b-a867d3d6498f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:55.125128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aa9f78-fcc6-4f47-bf5b-a867d3d6498f service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:55.144818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 707/1379] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:44:57.514486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc5dc8-1227-4c8b-a52d-5b7ea55e4939 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:57.515239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc5dc8-1227-4c8b-a52d-5b7ea55e4939 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:57.515239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc5dc8-1227-4c8b-a52d-5b7ea55e4939 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:57.521542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6b6d0-b0be-44db-93da-5525d56046f2 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:57.522146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6b6d0-b0be-44db-93da-5525d56046f2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:57.522146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6b6d0-b0be-44db-93da-5525d56046f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:57.529141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 673/1380] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:44:57 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:44:57.597327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-fdd6b6d0-b0be-44db-93da-5525d56046f2 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:44:57.598629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fdd6b6d0-b0be-44db-93da-5525d56046f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:57.701890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:57.708970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 708/1381] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:44:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:44:58.075746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbc5dc8-1227-4c8b-a52d-5b7ea55e4939 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:44:58.077321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cdbc5dc8-1227-4c8b-a52d-5b7ea55e4939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:58.239556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:58.251765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbc5dc8-1227-4c8b-a52d-5b7ea55e4939 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:44:58.369443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7437c7-66aa-4184-8b01-4600efc59c82 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:44:58.370163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7437c7-66aa-4184-8b01-4600efc59c82 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:44:58.370576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7437c7-66aa-4184-8b01-4600efc59c82 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:44:58.370854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7437c7-66aa-4184-8b01-4600efc59c82 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:44:58.372516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5d7437c7-66aa-4184-8b01-4600efc59c82 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:44:58.479804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7437c7-66aa-4184-8b01-4600efc59c82 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:44:58.481453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7437c7-66aa-4184-8b01-4600efc59c82 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:44:58.492071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:44:58.501827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 674/1382] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:44:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:01.433667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049748a-0547-440e-89c6-af265286cc94 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:01.434213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049748a-0547-440e-89c6-af265286cc94 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:01.434213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049748a-0547-440e-89c6-af265286cc94 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:01.435558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 709/1383] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:45:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:01.441573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b09f8c-366d-42e9-97c7-e739d75df023 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:01.441984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b09f8c-366d-42e9-97c7-e739d75df023 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:01.442251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b09f8c-366d-42e9-97c7-e739d75df023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:01.521969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-60b09f8c-366d-42e9-97c7-e739d75df023 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:01.523143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-60b09f8c-366d-42e9-97c7-e739d75df023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:01.644712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 675/1384] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:45:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:01.944575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c049748a-0547-440e-89c6-af265286cc94 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:01.946101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c049748a-0547-440e-89c6-af265286cc94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:02.131397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c049748a-0547-440e-89c6-af265286cc94 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:02.228985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:02.277677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d502a-19b4-45dc-b789-eba4fdba6ec3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:02.278292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d502a-19b4-45dc-b789-eba4fdba6ec3 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:02.278606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d502a-19b4-45dc-b789-eba4fdba6ec3 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:02.278887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d502a-19b4-45dc-b789-eba4fdba6ec3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:02.280548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-be8d502a-19b4-45dc-b789-eba4fdba6ec3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:02.377729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d502a-19b4-45dc-b789-eba4fdba6ec3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:02.379405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d502a-19b4-45dc-b789-eba4fdba6ec3 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:02.398447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 710/1385] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:04.633421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c212ed-6597-4634-8eef-e5bb6bd5b265 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:04.633957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c212ed-6597-4634-8eef-e5bb6bd5b265 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:04.633957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c212ed-6597-4634-8eef-e5bb6bd5b265 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:04.635475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 676/1386] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:45:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:04.640927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae9250b-b398-4b15-8771-271a7669b8c3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:04.641405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae9250b-b398-4b15-8771-271a7669b8c3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:04.641602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae9250b-b398-4b15-8771-271a7669b8c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:04.711848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-aae9250b-b398-4b15-8771-271a7669b8c3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:04.713082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-aae9250b-b398-4b15-8771-271a7669b8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:04.750782 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:04.817279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 711/1387] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:45:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:05.126796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c212ed-6597-4634-8eef-e5bb6bd5b265 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:05.128633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b3c212ed-6597-4634-8eef-e5bb6bd5b265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:05.208876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:05.281864 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c212ed-6597-4634-8eef-e5bb6bd5b265 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:05.399178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832039b0-0f9a-448d-85b8-7c89fcc6ea97 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:05.399793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832039b0-0f9a-448d-85b8-7c89fcc6ea97 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:05.400081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832039b0-0f9a-448d-85b8-7c89fcc6ea97 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:05.400343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832039b0-0f9a-448d-85b8-7c89fcc6ea97 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:05.402038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-832039b0-0f9a-448d-85b8-7c89fcc6ea97 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:05.442136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:05.496820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832039b0-0f9a-448d-85b8-7c89fcc6ea97 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:05.498461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832039b0-0f9a-448d-85b8-7c89fcc6ea97 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:05.519213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 677/1388] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:05.606811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eca485-10a7-42de-8af1-2da99dfdd4c4 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:05.607185 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eca485-10a7-42de-8af1-2da99dfdd4c4 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:05.607530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eca485-10a7-42de-8af1-2da99dfdd4c4 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:05.609031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 712/1389] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:45:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:05.614587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8052be-e259-453e-879e-1481b2fbe9ef None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:05.614890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8052be-e259-453e-879e-1481b2fbe9ef None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:05.615164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8052be-e259-453e-879e-1481b2fbe9ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:05.686326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0e8052be-e259-453e-879e-1481b2fbe9ef None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:05.687480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0e8052be-e259-453e-879e-1481b2fbe9ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:05.799540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 678/1390] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:45:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:06.103571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595e8678-2db0-4e50-ab52-96d2953e8517 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:06.105294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595e8678-2db0-4e50-ab52-96d2953e8517 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:06.105294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595e8678-2db0-4e50-ab52-96d2953e8517 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:06.105535 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 713/1391] 10.222.0.27 () {54 vars in 962 bytes} [Tue Oct 4 08:45:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:45:06.110915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34319c69-4db5-4f5a-a7ab-beb21c055f43 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:06.111203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34319c69-4db5-4f5a-a7ab-beb21c055f43 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:06.111480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34319c69-4db5-4f5a-a7ab-beb21c055f43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:06.127643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:06.190693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-34319c69-4db5-4f5a-a7ab-beb21c055f43 None None] MFA Rules not processed for user `146238397f514eada174ab3374be5418`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:06.192141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-34319c69-4db5-4f5a-a7ab-beb21c055f43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:06.270218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 679/1392] 10.222.0.27 () {58 vars in 1059 bytes} [Tue Oct 4 08:45:06 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:06.278898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595e8678-2db0-4e50-ab52-96d2953e8517 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:06.280417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-595e8678-2db0-4e50-ab52-96d2953e8517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:06.389427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595e8678-2db0-4e50-ab52-96d2953e8517 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:06.449164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:06.465069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e15ab7-62c1-4fa5-acc7-148e45f924a7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:06.465860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e15ab7-62c1-4fa5-acc7-148e45f924a7 service glance] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:06.465961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e15ab7-62c1-4fa5-acc7-148e45f924a7 service glance] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:06.466190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e15ab7-62c1-4fa5-acc7-148e45f924a7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:06.467852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-80e15ab7-62c1-4fa5-acc7-148e45f924a7 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:06.580521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e15ab7-62c1-4fa5-acc7-148e45f924a7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:06.581878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e15ab7-62c1-4fa5-acc7-148e45f924a7 service glance] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:06.601634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 714/1393] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:07.498704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f82de8-8726-4717-b2d4-e34a4c6d8a41 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:07.499231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f82de8-8726-4717-b2d4-e34a4c6d8a41 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:07.499231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f82de8-8726-4717-b2d4-e34a4c6d8a41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:07.567278 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-09f82de8-8726-4717-b2d4-e34a4c6d8a41 None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:07.568444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-09f82de8-8726-4717-b2d4-e34a4c6d8a41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:07.628281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:07.661334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 680/1394] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:45:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:07.705488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc91ff-f2da-4762-ba61-e7c7c3079f05 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:07.706001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc91ff-f2da-4762-ba61-e7c7c3079f05 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:07.706001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc91ff-f2da-4762-ba61-e7c7c3079f05 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:07.707438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 715/1395] 10.222.0.27 () {54 vars in 943 bytes} [Tue Oct 4 08:45:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:07.714306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e2201-64e3-4717-b278-9acf350a1acd None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:07.714732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e2201-64e3-4717-b278-9acf350a1acd None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:07.715108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e2201-64e3-4717-b278-9acf350a1acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:07.802223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d09e2201-64e3-4717-b278-9acf350a1acd None None] MFA Rules not processed for user `085a0cc3f5cb477989e6e75adea7c53a`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:07.803457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d09e2201-64e3-4717-b278-9acf350a1acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:07.875902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd4537-c82a-4e6c-a544-2d3df5f0ab55 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:07.876094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd4537-c82a-4e6c-a544-2d3df5f0ab55 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:07.876350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd4537-c82a-4e6c-a544-2d3df5f0ab55 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:07.878125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 716/1396] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:45:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:45:07.882979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:07.883259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:07.883533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:07.911872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 681/1397] 10.222.0.27 () {58 vars in 1040 bytes} [Tue Oct 4 08:45:07 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:07.921711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e2201-64e3-4717-b278-9acf350a1acd None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:07.931215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e2201-64e3-4717-b278-9acf350a1acd None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:07.956195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:07.957399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:07.959170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:07.999151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:08.039203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4e73ae-9feb-4de6-a6b0-c8bc652f17fc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:08.039821 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e73ae-9feb-4de6-a6b0-c8bc652f17fc service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:08.040096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e73ae-9feb-4de6-a6b0-c8bc652f17fc service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:08.040363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4e73ae-9feb-4de6-a6b0-c8bc652f17fc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:08.042777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3c4e73ae-9feb-4de6-a6b0-c8bc652f17fc service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:08.059022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 717/1398] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:45:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:45:08.126254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4e73ae-9feb-4de6-a6b0-c8bc652f17fc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:08.128063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4e73ae-9feb-4de6-a6b0-c8bc652f17fc service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:08.137081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 682/1399] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:45:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 08:45:08.352872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:08.354767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:08.475709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962d1a24-e852-44b6-8ceb-cebc759042a5 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:08.531664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:08.595288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fa8510-10d2-4787-bdde-5fca3f700e83 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:08.595901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa8510-10d2-4787-bdde-5fca3f700e83 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:08.596336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa8510-10d2-4787-bdde-5fca3f700e83 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:08.596465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa8510-10d2-4787-bdde-5fca3f700e83 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:08.598638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e3fa8510-10d2-4787-bdde-5fca3f700e83 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:08.707427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fa8510-10d2-4787-bdde-5fca3f700e83 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:08.709822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fa8510-10d2-4787-bdde-5fca3f700e83 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:08.731189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 718/1400] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:11.100331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b895af7-50ae-45d2-9aef-9af97a0f4c22 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:11.100941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b895af7-50ae-45d2-9aef-9af97a0f4c22 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:11.100941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b895af7-50ae-45d2-9aef-9af97a0f4c22 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:11.102378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 683/1401] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:45:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:11.108484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7f34d6-44ad-48d2-9172-14e9f3d2bb6b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:11.109598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7f34d6-44ad-48d2-9172-14e9f3d2bb6b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:11.109898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7f34d6-44ad-48d2-9172-14e9f3d2bb6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:11.190690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8c7f34d6-44ad-48d2-9172-14e9f3d2bb6b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:11.192274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8c7f34d6-44ad-48d2-9172-14e9f3d2bb6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:11.209426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:11.299587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 719/1402] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:45:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:11.607715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b895af7-50ae-45d2-9aef-9af97a0f4c22 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:11.609505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1b895af7-50ae-45d2-9aef-9af97a0f4c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:11.679139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:11.775805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b895af7-50ae-45d2-9aef-9af97a0f4c22 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:11.905077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e06f125-2411-45e6-b732-df20e77cf7d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:11.905752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06f125-2411-45e6-b732-df20e77cf7d9 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:11.906152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06f125-2411-45e6-b732-df20e77cf7d9 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:11.906454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06f125-2411-45e6-b732-df20e77cf7d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:11.908047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8e06f125-2411-45e6-b732-df20e77cf7d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:11.933960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:12.140176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e06f125-2411-45e6-b732-df20e77cf7d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:12.141782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e06f125-2411-45e6-b732-df20e77cf7d9 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:12.164886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 684/1403] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 557 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:14.539942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3dc6d6-202c-4a3a-b5fe-be1eb3311523 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:14.540507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3dc6d6-202c-4a3a-b5fe-be1eb3311523 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:14.540507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3dc6d6-202c-4a3a-b5fe-be1eb3311523 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:14.542185 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 720/1404] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:45:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:14.547414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b5a5e7-364e-4059-985b-d67047577490 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:14.547860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b5a5e7-364e-4059-985b-d67047577490 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:14.547948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b5a5e7-364e-4059-985b-d67047577490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:14.626712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b7b5a5e7-364e-4059-985b-d67047577490 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:14.628192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b7b5a5e7-364e-4059-985b-d67047577490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:14.735075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:14.741037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 685/1405] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:45:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:15.042880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3dc6d6-202c-4a3a-b5fe-be1eb3311523 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:15.044508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6e3dc6d6-202c-4a3a-b5fe-be1eb3311523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:15.239764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:15.248575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3dc6d6-202c-4a3a-b5fe-be1eb3311523 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:15.390825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af6fd0a-3a13-454a-ac61-e7dcf7cf4865 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:15.391564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6fd0a-3a13-454a-ac61-e7dcf7cf4865 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:15.391842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6fd0a-3a13-454a-ac61-e7dcf7cf4865 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:15.392078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af6fd0a-3a13-454a-ac61-e7dcf7cf4865 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:15.393744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6af6fd0a-3a13-454a-ac61-e7dcf7cf4865 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:15.520292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af6fd0a-3a13-454a-ac61-e7dcf7cf4865 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:15.522301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af6fd0a-3a13-454a-ac61-e7dcf7cf4865 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:15.564718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 721/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:18.408197 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85b6f51-ee89-4a3e-8eda-fb47af4364d6 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:18.408803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85b6f51-ee89-4a3e-8eda-fb47af4364d6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:18.408803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85b6f51-ee89-4a3e-8eda-fb47af4364d6 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:18.410243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 686/1407] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:45:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:18.415282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19aae37d-4675-499d-be27-45ea6a2f923a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:18.415690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19aae37d-4675-499d-be27-45ea6a2f923a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:18.415826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19aae37d-4675-499d-be27-45ea6a2f923a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:18.420484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:18.492173 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-19aae37d-4675-499d-be27-45ea6a2f923a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:18.495739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-19aae37d-4675-499d-be27-45ea6a2f923a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:18.607861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 722/1408] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:45:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:18.901488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85b6f51-ee89-4a3e-8eda-fb47af4364d6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:18.903051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d85b6f51-ee89-4a3e-8eda-fb47af4364d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:19.070136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85b6f51-ee89-4a3e-8eda-fb47af4364d6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:19.143618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:19.196121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607a4080-95a7-4e7a-ab5c-b19abb180d23 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:19.196747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a4080-95a7-4e7a-ab5c-b19abb180d23 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:19.197023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a4080-95a7-4e7a-ab5c-b19abb180d23 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:19.197294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a4080-95a7-4e7a-ab5c-b19abb180d23 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:19.199095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-607a4080-95a7-4e7a-ab5c-b19abb180d23 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:19.307537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607a4080-95a7-4e7a-ab5c-b19abb180d23 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:19.309196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607a4080-95a7-4e7a-ab5c-b19abb180d23 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:19.329187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 687/1409] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:28.606139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=64 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:29.326478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=64 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:32.959374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02509bcf-dd80-467c-b9a2-d93077e08d48 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:32.959902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02509bcf-dd80-467c-b9a2-d93077e08d48 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:32.959902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02509bcf-dd80-467c-b9a2-d93077e08d48 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:32.961168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 723/1410] 10.222.0.27 () {54 vars in 918 bytes} [Tue Oct 4 08:45:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:32.966258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c826cbc3-a0b6-4659-9942-d1cc4642c63d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:32.966707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c826cbc3-a0b6-4659-9942-d1cc4642c63d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:32.966830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c826cbc3-a0b6-4659-9942-d1cc4642c63d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:33.048703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c826cbc3-a0b6-4659-9942-d1cc4642c63d None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:33.049935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c826cbc3-a0b6-4659-9942-d1cc4642c63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:33.157882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 688/1411] 10.222.0.27 () {58 vars in 1015 bytes} [Tue Oct 4 08:45:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:33.172928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e8321a-9657-4de5-8a60-d3286772b306 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:33.173238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e8321a-9657-4de5-8a60-d3286772b306 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:33.173509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e8321a-9657-4de5-8a60-d3286772b306 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:33.174862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 724/1412] 10.222.0.27 () {54 vars in 918 bytes} [Tue Oct 4 08:45:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:33.179456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e928c092-e034-4ce4-9643-cdb3d174b6d5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:33.179737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e928c092-e034-4ce4-9643-cdb3d174b6d5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:33.180036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e928c092-e034-4ce4-9643-cdb3d174b6d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:33.252496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:33.255069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e928c092-e034-4ce4-9643-cdb3d174b6d5 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:33.256279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e928c092-e034-4ce4-9643-cdb3d174b6d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:33.375900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 689/1413] 10.222.0.27 () {58 vars in 1015 bytes} [Tue Oct 4 08:45:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:36.035728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c174527-3272-4ccb-9bde-8431a08fc736 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:36.036163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c174527-3272-4ccb-9bde-8431a08fc736 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:36.036331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c174527-3272-4ccb-9bde-8431a08fc736 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:36.037698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 725/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:45:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:36.044173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24dde0e-69f8-4e36-b738-de68eec4c4f0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:36.047710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24dde0e-69f8-4e36-b738-de68eec4c4f0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:36.047710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24dde0e-69f8-4e36-b738-de68eec4c4f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:36.118103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f24dde0e-69f8-4e36-b738-de68eec4c4f0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:36.120088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f24dde0e-69f8-4e36-b738-de68eec4c4f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:36.175540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:36.217705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 690/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:45:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:36.511494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c174527-3272-4ccb-9bde-8431a08fc736 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:36.513015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9c174527-3272-4ccb-9bde-8431a08fc736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:36.657181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c174527-3272-4ccb-9bde-8431a08fc736 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:36.728200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:36.768455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a4314d-a122-4348-8245-b750982992a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:36.769052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a4314d-a122-4348-8245-b750982992a2 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:36.769326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a4314d-a122-4348-8245-b750982992a2 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:36.769594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a4314d-a122-4348-8245-b750982992a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:36.771211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-26a4314d-a122-4348-8245-b750982992a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:36.859927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a4314d-a122-4348-8245-b750982992a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:36.861512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a4314d-a122-4348-8245-b750982992a2 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:36.879996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 726/1416] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:45:38.029148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a6b40d-a9c7-49d0-9a11-0f106baa80d6 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:38.029717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a6b40d-a9c7-49d0-9a11-0f106baa80d6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:38.029717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a6b40d-a9c7-49d0-9a11-0f106baa80d6 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:38.031040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 691/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:38.037196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13da66-2172-4bb3-92b7-24748db0d41c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:38.037601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13da66-2172-4bb3-92b7-24748db0d41c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:38.037746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13da66-2172-4bb3-92b7-24748db0d41c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:38.116490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8e13da66-2172-4bb3-92b7-24748db0d41c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:38.117709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8e13da66-2172-4bb3-92b7-24748db0d41c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:38.145853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:38.215199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 727/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:38.373651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a6b40d-a9c7-49d0-9a11-0f106baa80d6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:38.376535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-07a6b40d-a9c7-49d0-9a11-0f106baa80d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:38.524411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a6b40d-a9c7-49d0-9a11-0f106baa80d6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:38.563380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:38.633845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b03451-6adb-4834-8b76-335bd69a8ab9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:38.634376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b03451-6adb-4834-8b76-335bd69a8ab9 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:38.634793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b03451-6adb-4834-8b76-335bd69a8ab9 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:38.635552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b03451-6adb-4834-8b76-335bd69a8ab9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:38.637166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-43b03451-6adb-4834-8b76-335bd69a8ab9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:38.725585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b03451-6adb-4834-8b76-335bd69a8ab9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:38.727193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b03451-6adb-4834-8b76-335bd69a8ab9 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:38.745126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 692/1419] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:39.878580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b5db34-9831-424e-a40b-2a6340fcc890 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:39.879102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b5db34-9831-424e-a40b-2a6340fcc890 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:39.879102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b5db34-9831-424e-a40b-2a6340fcc890 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:39.880352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 728/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:39.887413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73389090-9c8a-45cc-a259-9e1832617796 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:39.887907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73389090-9c8a-45cc-a259-9e1832617796 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:39.887907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73389090-9c8a-45cc-a259-9e1832617796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:39.942006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:39.963705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-73389090-9c8a-45cc-a259-9e1832617796 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:39.964929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-73389090-9c8a-45cc-a259-9e1832617796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:40.060194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 693/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:40.212291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b5db34-9831-424e-a40b-2a6340fcc890 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:40.213735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c2b5db34-9831-424e-a40b-2a6340fcc890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:40.358586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b5db34-9831-424e-a40b-2a6340fcc890 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:40.371063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:40.474128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be95a421-4d8f-4f61-ae54-f3d5acf93267 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:40.474698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95a421-4d8f-4f61-ae54-f3d5acf93267 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:40.474975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95a421-4d8f-4f61-ae54-f3d5acf93267 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:40.475236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95a421-4d8f-4f61-ae54-f3d5acf93267 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:40.476863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-be95a421-4d8f-4f61-ae54-f3d5acf93267 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:40.566684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be95a421-4d8f-4f61-ae54-f3d5acf93267 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:40.568271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be95a421-4d8f-4f61-ae54-f3d5acf93267 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:40.585953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 729/1422] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:41.694293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d137536-c338-4c2a-82cc-7b57221679e7 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:41.694838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d137536-c338-4c2a-82cc-7b57221679e7 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:41.694838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d137536-c338-4c2a-82cc-7b57221679e7 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:41.696134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 694/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:41.702856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da11fc4-68b8-4493-8f32-dd69134082f9 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:41.703300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da11fc4-68b8-4493-8f32-dd69134082f9 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:41.703415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da11fc4-68b8-4493-8f32-dd69134082f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:41.727546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:41.784917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1da11fc4-68b8-4493-8f32-dd69134082f9 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:41.786124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1da11fc4-68b8-4493-8f32-dd69134082f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:41.882358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 730/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:42.035361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d137536-c338-4c2a-82cc-7b57221679e7 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:42.037011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8d137536-c338-4c2a-82cc-7b57221679e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:42.149816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:42.186690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d137536-c338-4c2a-82cc-7b57221679e7 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:42.301691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6502094-9b84-4295-a604-16aeb7d0efde service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:42.302213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6502094-9b84-4295-a604-16aeb7d0efde service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:42.302514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6502094-9b84-4295-a604-16aeb7d0efde service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:42.302785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6502094-9b84-4295-a604-16aeb7d0efde service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:42.304383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a6502094-9b84-4295-a604-16aeb7d0efde service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:42.387794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:42.404692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6502094-9b84-4295-a604-16aeb7d0efde service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:42.407317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6502094-9b84-4295-a604-16aeb7d0efde service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:42.428376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 695/1425] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:43.545126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16e2d64-df62-4442-bac6-69d9cb06fe10 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:43.545639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16e2d64-df62-4442-bac6-69d9cb06fe10 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:43.545639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16e2d64-df62-4442-bac6-69d9cb06fe10 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:43.546997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 731/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:43.553153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d63bb7-7eb5-413e-90d6-4a6a674d4a11 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:43.562811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d63bb7-7eb5-413e-90d6-4a6a674d4a11 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:43.562811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d63bb7-7eb5-413e-90d6-4a6a674d4a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:43.631891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-32d63bb7-7eb5-413e-90d6-4a6a674d4a11 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:43.633093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-32d63bb7-7eb5-413e-90d6-4a6a674d4a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:43.729073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 696/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:43.880482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16e2d64-df62-4442-bac6-69d9cb06fe10 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:43.882066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c16e2d64-df62-4442-bac6-69d9cb06fe10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:43.964739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:44.026860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16e2d64-df62-4442-bac6-69d9cb06fe10 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:44.137340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff05696-bed7-4e3f-ad2d-30b4bb329068 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:44.137859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05696-bed7-4e3f-ad2d-30b4bb329068 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:44.138131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05696-bed7-4e3f-ad2d-30b4bb329068 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:44.138394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05696-bed7-4e3f-ad2d-30b4bb329068 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:44.140002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9ff05696-bed7-4e3f-ad2d-30b4bb329068 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:44.190106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:44.230502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff05696-bed7-4e3f-ad2d-30b4bb329068 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:44.232149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff05696-bed7-4e3f-ad2d-30b4bb329068 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:44.250067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 732/1428] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:45.422339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63943872-09df-4d0c-beba-96ff8e74def3 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:45.422841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63943872-09df-4d0c-beba-96ff8e74def3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:45.422966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63943872-09df-4d0c-beba-96ff8e74def3 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:45.424297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 697/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:45.430722 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f408774-93d9-4364-ad51-25a4e70bd3e8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:45.431138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f408774-93d9-4364-ad51-25a4e70bd3e8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:45.431275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f408774-93d9-4364-ad51-25a4e70bd3e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:45.506484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9f408774-93d9-4364-ad51-25a4e70bd3e8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:45.507726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9f408774-93d9-4364-ad51-25a4e70bd3e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:45.602447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 733/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:45.761986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63943872-09df-4d0c-beba-96ff8e74def3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:45.763565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-63943872-09df-4d0c-beba-96ff8e74def3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:45.811493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:45.906547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63943872-09df-4d0c-beba-96ff8e74def3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:46.017397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d45a04-c1fa-4d44-818c-478cb5d5ab9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:46.017993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d45a04-c1fa-4d44-818c-478cb5d5ab9f service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:46.018265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d45a04-c1fa-4d44-818c-478cb5d5ab9f service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:46.018568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d45a04-c1fa-4d44-818c-478cb5d5ab9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:46.020145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e9d45a04-c1fa-4d44-818c-478cb5d5ab9f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:46.029216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:46.112593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d45a04-c1fa-4d44-818c-478cb5d5ab9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:46.114166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d45a04-c1fa-4d44-818c-478cb5d5ab9f service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:46.132437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 698/1431] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:47.234892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775a0e82-6826-44e8-a58e-acd84a2e2561 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:47.235478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775a0e82-6826-44e8-a58e-acd84a2e2561 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:47.235478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775a0e82-6826-44e8-a58e-acd84a2e2561 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:47.236766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 734/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:47.242984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2d206-6070-46fe-9659-0fd107a0f294 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:47.243441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2d206-6070-46fe-9659-0fd107a0f294 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:47.243534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2d206-6070-46fe-9659-0fd107a0f294 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:47.318709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7ba2d206-6070-46fe-9659-0fd107a0f294 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:47.319901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7ba2d206-6070-46fe-9659-0fd107a0f294 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:47.387526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:47.417682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 699/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:47.571435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775a0e82-6826-44e8-a58e-acd84a2e2561 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:47.573423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-775a0e82-6826-44e8-a58e-acd84a2e2561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:47.590574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:47.720026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775a0e82-6826-44e8-a58e-acd84a2e2561 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:47.812097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:47.838018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f5667d-780a-467a-8d32-bf778d5661a4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:47.838581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f5667d-780a-467a-8d32-bf778d5661a4 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:47.838864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f5667d-780a-467a-8d32-bf778d5661a4 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:47.839132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f5667d-780a-467a-8d32-bf778d5661a4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:47.841437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b2f5667d-780a-467a-8d32-bf778d5661a4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:47.931096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f5667d-780a-467a-8d32-bf778d5661a4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:47.933264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f5667d-780a-467a-8d32-bf778d5661a4 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:47.950605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 735/1434] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:49.083689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b593b0-306c-4a87-9937-95370a0fe1f7 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:49.084250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b593b0-306c-4a87-9937-95370a0fe1f7 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:49.084250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b593b0-306c-4a87-9937-95370a0fe1f7 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:49.085592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 700/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:49.092416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1416f98f-a29d-455e-b524-e38d93dc1d1b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:49.092858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1416f98f-a29d-455e-b524-e38d93dc1d1b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:49.093019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1416f98f-a29d-455e-b524-e38d93dc1d1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:49.168408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1416f98f-a29d-455e-b524-e38d93dc1d1b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:49.169609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1416f98f-a29d-455e-b524-e38d93dc1d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:49.209450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:49.263358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 736/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:49.416550 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b593b0-306c-4a87-9937-95370a0fe1f7 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:49.418081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-29b593b0-306c-4a87-9937-95370a0fe1f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:49.561711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b593b0-306c-4a87-9937-95370a0fe1f7 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:49.616166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:49.674074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e914ab-d197-4484-a620-62894a64b4a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:49.674630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e914ab-d197-4484-a620-62894a64b4a1 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:49.674910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e914ab-d197-4484-a620-62894a64b4a1 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:49.675177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e914ab-d197-4484-a620-62894a64b4a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:49.676776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-52e914ab-d197-4484-a620-62894a64b4a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:49.767383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e914ab-d197-4484-a620-62894a64b4a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:49.768977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e914ab-d197-4484-a620-62894a64b4a1 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:49.793389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 701/1437] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:50.909806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09b77a-b3ac-4b38-a7c3-407d2a0d7c11 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:50.910376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09b77a-b3ac-4b38-a7c3-407d2a0d7c11 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:50.910376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09b77a-b3ac-4b38-a7c3-407d2a0d7c11 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:50.911711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 737/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:45:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:50.918105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a485f-825f-45e6-a82d-60ced5d02dfc None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:50.918591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a485f-825f-45e6-a82d-60ced5d02dfc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:50.918696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4a485f-825f-45e6-a82d-60ced5d02dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:50.993380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5d4a485f-825f-45e6-a82d-60ced5d02dfc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:50.994867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5d4a485f-825f-45e6-a82d-60ced5d02dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:50.996873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:51.094474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 702/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:45:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:51.242779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09b77a-b3ac-4b38-a7c3-407d2a0d7c11 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:51.244799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2b09b77a-b3ac-4b38-a7c3-407d2a0d7c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:51.392838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09b77a-b3ac-4b38-a7c3-407d2a0d7c11 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:51.419857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:51.505243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874e1e50-7eab-405a-bcc0-cafd4997f049 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:51.505751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874e1e50-7eab-405a-bcc0-cafd4997f049 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:51.506269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874e1e50-7eab-405a-bcc0-cafd4997f049 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:51.506580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874e1e50-7eab-405a-bcc0-cafd4997f049 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:51.508166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-874e1e50-7eab-405a-bcc0-cafd4997f049 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:51.598009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874e1e50-7eab-405a-bcc0-cafd4997f049 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:51.599633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874e1e50-7eab-405a-bcc0-cafd4997f049 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:51.617460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 738/1440] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:53.707176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6e14e7-8dc7-4da7-9e4f-c38ec141d8b5 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:53.707731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6e14e7-8dc7-4da7-9e4f-c38ec141d8b5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:53.707731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6e14e7-8dc7-4da7-9e4f-c38ec141d8b5 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:53.708968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 703/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:45:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:53.714392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7ba0be-019c-446d-a25f-36f6e61cd7a8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:53.714851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7ba0be-019c-446d-a25f-36f6e61cd7a8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:53.715008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7ba0be-019c-446d-a25f-36f6e61cd7a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:53.755261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:53.780769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ea7ba0be-019c-446d-a25f-36f6e61cd7a8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:53.781967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ea7ba0be-019c-446d-a25f-36f6e61cd7a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:53.879985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 739/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:45:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:45:54.166492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6e14e7-8dc7-4da7-9e4f-c38ec141d8b5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:54.168074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7a6e14e7-8dc7-4da7-9e4f-c38ec141d8b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:54.272710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6e14e7-8dc7-4da7-9e4f-c38ec141d8b5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:54.303985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:54.344767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c99dff-b016-4070-ba12-fa6d8a69d554 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:54.345365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c99dff-b016-4070-ba12-fa6d8a69d554 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:54.345639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c99dff-b016-4070-ba12-fa6d8a69d554 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:54.345908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c99dff-b016-4070-ba12-fa6d8a69d554 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:54.347540 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-48c99dff-b016-4070-ba12-fa6d8a69d554 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:54.457352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c99dff-b016-4070-ba12-fa6d8a69d554 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:54.458747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c99dff-b016-4070-ba12-fa6d8a69d554 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:54.476180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 704/1443] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:45:56.786003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c8e2a8-d83a-4bb1-ba85-2130457430c8 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:56.786605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c8e2a8-d83a-4bb1-ba85-2130457430c8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:56.786605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c8e2a8-d83a-4bb1-ba85-2130457430c8 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:56.787896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 740/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:45:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:45:56.800096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6387f8a-4018-4a93-afd6-af92a2b5356e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:56.800603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6387f8a-4018-4a93-afd6-af92a2b5356e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:56.800707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6387f8a-4018-4a93-afd6-af92a2b5356e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:56.865862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f6387f8a-4018-4a93-afd6-af92a2b5356e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:45:56.867093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f6387f8a-4018-4a93-afd6-af92a2b5356e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:56.880398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:56.963237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 705/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:45:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:45:57.251056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c8e2a8-d83a-4bb1-ba85-2130457430c8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:45:57.252876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-28c8e2a8-d83a-4bb1-ba85-2130457430c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:57.358762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c8e2a8-d83a-4bb1-ba85-2130457430c8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:45:57.371767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:45:57.433448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373884a3-c340-43fd-91ed-602742d2ae21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:45:57.434052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373884a3-c340-43fd-91ed-602742d2ae21 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:45:57.434329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373884a3-c340-43fd-91ed-602742d2ae21 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:45:57.434626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373884a3-c340-43fd-91ed-602742d2ae21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:45:57.436222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-373884a3-c340-43fd-91ed-602742d2ae21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:45:57.546601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373884a3-c340-43fd-91ed-602742d2ae21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:45:57.547966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373884a3-c340-43fd-91ed-602742d2ae21 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:45:57.565711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 741/1446] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:45:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:00.114025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34092615-cfa6-4f51-9257-d498949f9463 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:00.114644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34092615-cfa6-4f51-9257-d498949f9463 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:00.114644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34092615-cfa6-4f51-9257-d498949f9463 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:00.115931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 706/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:00.121727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d516f51-96e5-4adc-9769-ed8d154bdafc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:00.122132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d516f51-96e5-4adc-9769-ed8d154bdafc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:00.122269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d516f51-96e5-4adc-9769-ed8d154bdafc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:00.170564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:00.187993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1d516f51-96e5-4adc-9769-ed8d154bdafc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:00.189211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1d516f51-96e5-4adc-9769-ed8d154bdafc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:00.284987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 742/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:00.567638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34092615-cfa6-4f51-9257-d498949f9463 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:00.569254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-34092615-cfa6-4f51-9257-d498949f9463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:00.672775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34092615-cfa6-4f51-9257-d498949f9463 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:00.713173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:00.746279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d2e50a-b4bc-4386-8ff9-ff3580f28343 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:00.746930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d2e50a-b4bc-4386-8ff9-ff3580f28343 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:00.747209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d2e50a-b4bc-4386-8ff9-ff3580f28343 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:00.747481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d2e50a-b4bc-4386-8ff9-ff3580f28343 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:00.749079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-48d2e50a-b4bc-4386-8ff9-ff3580f28343 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:00.856966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d2e50a-b4bc-4386-8ff9-ff3580f28343 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:00.858491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d2e50a-b4bc-4386-8ff9-ff3580f28343 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:00.876394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 707/1449] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:03.145687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6a6d1-c980-410e-b1d3-22d9e8e5e4a8 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:03.146231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6a6d1-c980-410e-b1d3-22d9e8e5e4a8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:03.146231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6a6d1-c980-410e-b1d3-22d9e8e5e4a8 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:03.147641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 743/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:03.152954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5ab749-5498-46b3-bebb-5715c2ba000c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:03.153391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5ab749-5498-46b3-bebb-5715c2ba000c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:03.153499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5ab749-5498-46b3-bebb-5715c2ba000c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:03.218808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0b5ab749-5498-46b3-bebb-5715c2ba000c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:03.220021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0b5ab749-5498-46b3-bebb-5715c2ba000c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:03.241070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:03.316693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 708/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:03.608519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6a6d1-c980-410e-b1d3-22d9e8e5e4a8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:03.610068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3ae6a6d1-c980-410e-b1d3-22d9e8e5e4a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:03.712120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6a6d1-c980-410e-b1d3-22d9e8e5e4a8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:03.732919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:03.784989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1305bb2f-b96a-40c2-91f5-03be4ca0e9cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:03.785609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305bb2f-b96a-40c2-91f5-03be4ca0e9cb service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:03.785884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305bb2f-b96a-40c2-91f5-03be4ca0e9cb service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:03.786147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305bb2f-b96a-40c2-91f5-03be4ca0e9cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:03.787796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1305bb2f-b96a-40c2-91f5-03be4ca0e9cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:03.908440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1305bb2f-b96a-40c2-91f5-03be4ca0e9cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:03.909789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1305bb2f-b96a-40c2-91f5-03be4ca0e9cb service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:03.928029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 744/1452] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:46:06.141438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49b6dec-2c5d-4081-ac79-6b2ce37baa57 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:06.142042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49b6dec-2c5d-4081-ac79-6b2ce37baa57 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:06.142042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49b6dec-2c5d-4081-ac79-6b2ce37baa57 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:06.143437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 709/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:06.148990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cf4d62-83df-45b4-ba8f-b380ce8b9d00 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:06.149386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cf4d62-83df-45b4-ba8f-b380ce8b9d00 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:06.149539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cf4d62-83df-45b4-ba8f-b380ce8b9d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:06.218825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:06.221446 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e8cf4d62-83df-45b4-ba8f-b380ce8b9d00 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:06.222699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e8cf4d62-83df-45b4-ba8f-b380ce8b9d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:06.320324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 745/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:06.613179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49b6dec-2c5d-4081-ac79-6b2ce37baa57 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:06.614765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c49b6dec-2c5d-4081-ac79-6b2ce37baa57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:06.718006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49b6dec-2c5d-4081-ac79-6b2ce37baa57 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:06.767746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:06.791745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a4d297-5674-45f5-ab61-f70f929a817d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:06.792345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4d297-5674-45f5-ab61-f70f929a817d service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:06.792620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4d297-5674-45f5-ab61-f70f929a817d service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:06.792887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4d297-5674-45f5-ab61-f70f929a817d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:06.794562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-53a4d297-5674-45f5-ab61-f70f929a817d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:06.902718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a4d297-5674-45f5-ab61-f70f929a817d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:06.904137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a4d297-5674-45f5-ab61-f70f929a817d service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:06.923619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 710/1455] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:09.122755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6cb8d9-e227-4392-94a3-0b3598737cd7 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:09.123325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6cb8d9-e227-4392-94a3-0b3598737cd7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:09.123325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6cb8d9-e227-4392-94a3-0b3598737cd7 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:09.124629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 746/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:09.130323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f78e3-056d-47fd-96e1-31c1c9f25d1c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:09.130745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f78e3-056d-47fd-96e1-31c1c9f25d1c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:09.130890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f78e3-056d-47fd-96e1-31c1c9f25d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:09.195658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-937f78e3-056d-47fd-96e1-31c1c9f25d1c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:09.197587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-937f78e3-056d-47fd-96e1-31c1c9f25d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:09.225811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:09.295058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 711/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:09.577963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6cb8d9-e227-4392-94a3-0b3598737cd7 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:09.579536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cd6cb8d9-e227-4392-94a3-0b3598737cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:09.689413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6cb8d9-e227-4392-94a3-0b3598737cd7 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:09.717267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:09.762117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff29c0c9-8040-4c77-a626-dd4c1b505498 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:09.762748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff29c0c9-8040-4c77-a626-dd4c1b505498 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:09.763026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff29c0c9-8040-4c77-a626-dd4c1b505498 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:09.763295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff29c0c9-8040-4c77-a626-dd4c1b505498 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:09.764881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ff29c0c9-8040-4c77-a626-dd4c1b505498 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:09.873312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff29c0c9-8040-4c77-a626-dd4c1b505498 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:09.874810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff29c0c9-8040-4c77-a626-dd4c1b505498 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:09.894249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 747/1458] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:12.109339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535e5b5a-6428-42a9-b2f7-317116487e13 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:12.109900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535e5b5a-6428-42a9-b2f7-317116487e13 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:12.109900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535e5b5a-6428-42a9-b2f7-317116487e13 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:12.111577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 712/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:12.116979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c83f6-7fd0-44e0-8d1d-753393e1f07f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:12.117396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c83f6-7fd0-44e0-8d1d-753393e1f07f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:12.117529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c83f6-7fd0-44e0-8d1d-753393e1f07f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:12.185963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d47c83f6-7fd0-44e0-8d1d-753393e1f07f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:12.187192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d47c83f6-7fd0-44e0-8d1d-753393e1f07f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:12.195422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:12.283061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 748/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:12.578098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535e5b5a-6428-42a9-b2f7-317116487e13 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:12.579712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-535e5b5a-6428-42a9-b2f7-317116487e13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:12.685708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535e5b5a-6428-42a9-b2f7-317116487e13 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:12.740559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:12.758853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f333db5-cce6-4d52-b27e-1287163bf0f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:12.760147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f333db5-cce6-4d52-b27e-1287163bf0f2 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:12.760423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f333db5-cce6-4d52-b27e-1287163bf0f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:12.760689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f333db5-cce6-4d52-b27e-1287163bf0f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:12.762313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9f333db5-cce6-4d52-b27e-1287163bf0f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:12.868819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f333db5-cce6-4d52-b27e-1287163bf0f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:12.870235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f333db5-cce6-4d52-b27e-1287163bf0f2 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:12.890236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 713/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:15.086006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025612a5-bc47-481c-ad3b-0bfda9e30fe1 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:15.086637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025612a5-bc47-481c-ad3b-0bfda9e30fe1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:15.086637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025612a5-bc47-481c-ad3b-0bfda9e30fe1 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:15.088680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 749/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:15.093910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d640b60-0651-4785-b8a2-df849b01865a None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:15.094320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d640b60-0651-4785-b8a2-df849b01865a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:15.094537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d640b60-0651-4785-b8a2-df849b01865a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:15.160083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8d640b60-0651-4785-b8a2-df849b01865a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:15.161277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8d640b60-0651-4785-b8a2-df849b01865a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:15.199200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:15.258060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 714/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:15.540831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025612a5-bc47-481c-ad3b-0bfda9e30fe1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:15.542353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-025612a5-bc47-481c-ad3b-0bfda9e30fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:15.646396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025612a5-bc47-481c-ad3b-0bfda9e30fe1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:15.685071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:15.719735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04be99e1-8c34-47f0-8d12-e80455f53ad1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:15.720341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04be99e1-8c34-47f0-8d12-e80455f53ad1 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:15.720618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04be99e1-8c34-47f0-8d12-e80455f53ad1 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:15.720889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04be99e1-8c34-47f0-8d12-e80455f53ad1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:15.722530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-04be99e1-8c34-47f0-8d12-e80455f53ad1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:15.828734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04be99e1-8c34-47f0-8d12-e80455f53ad1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:15.830167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04be99e1-8c34-47f0-8d12-e80455f53ad1 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:15.848069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 750/1464] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:18.034035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c40dfd2-c57f-4b58-8cda-4fc4c892c540 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:18.034662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c40dfd2-c57f-4b58-8cda-4fc4c892c540 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:18.034662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c40dfd2-c57f-4b58-8cda-4fc4c892c540 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:18.036024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 715/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:18.042239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c0858-58be-49ee-aa7b-46d9e53f36ff None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:18.042680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c0858-58be-49ee-aa7b-46d9e53f36ff None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:18.042820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c0858-58be-49ee-aa7b-46d9e53f36ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:18.107930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-493c0858-58be-49ee-aa7b-46d9e53f36ff None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:18.109129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-493c0858-58be-49ee-aa7b-46d9e53f36ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:18.127109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:18.203265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 751/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:18.487072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c40dfd2-c57f-4b58-8cda-4fc4c892c540 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:18.488628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6c40dfd2-c57f-4b58-8cda-4fc4c892c540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:18.591607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c40dfd2-c57f-4b58-8cda-4fc4c892c540 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:18.662692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:18.669372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231aa63f-e246-46db-b669-ade46f32db5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:18.674501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231aa63f-e246-46db-b669-ade46f32db5a service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:18.674501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231aa63f-e246-46db-b669-ade46f32db5a service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:18.674501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231aa63f-e246-46db-b669-ade46f32db5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:18.674501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-231aa63f-e246-46db-b669-ade46f32db5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:18.781717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231aa63f-e246-46db-b669-ade46f32db5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:18.783187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231aa63f-e246-46db-b669-ade46f32db5a service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:18.800804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 716/1467] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:20.997837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd3e4ff-9d85-4d0f-ac9d-8697d889ac50 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:20.998429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd3e4ff-9d85-4d0f-ac9d-8697d889ac50 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:20.998429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd3e4ff-9d85-4d0f-ac9d-8697d889ac50 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:20.999749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 752/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:46:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:21.005310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc121e57-1fbb-4974-81b8-7ab25edb6d82 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:21.005734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc121e57-1fbb-4974-81b8-7ab25edb6d82 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:21.005845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc121e57-1fbb-4974-81b8-7ab25edb6d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:21.071650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-bc121e57-1fbb-4974-81b8-7ab25edb6d82 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:21.072868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bc121e57-1fbb-4974-81b8-7ab25edb6d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:21.120683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:21.168410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 717/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:46:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:21.459519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd3e4ff-9d85-4d0f-ac9d-8697d889ac50 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:21.461067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4bd3e4ff-9d85-4d0f-ac9d-8697d889ac50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:21.565187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd3e4ff-9d85-4d0f-ac9d-8697d889ac50 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:21.615772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:21.642203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b476b0e4-0c64-4782-a5fa-01d330d4fe0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:21.642904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476b0e4-0c64-4782-a5fa-01d330d4fe0b service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:21.643175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476b0e4-0c64-4782-a5fa-01d330d4fe0b service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:21.643438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476b0e4-0c64-4782-a5fa-01d330d4fe0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:21.645021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b476b0e4-0c64-4782-a5fa-01d330d4fe0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:21.751839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476b0e4-0c64-4782-a5fa-01d330d4fe0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:21.753531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476b0e4-0c64-4782-a5fa-01d330d4fe0b service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:21.772132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 753/1470] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:31.166859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=22 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:31.769873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=66 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:42.015896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698d1fc6-b10f-4833-9c7a-ce0d49937d47 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:42.016469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698d1fc6-b10f-4833-9c7a-ce0d49937d47 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:42.016469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698d1fc6-b10f-4833-9c7a-ce0d49937d47 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:42.017776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 718/1471] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:42.023105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e82cad-5fa5-4b15-8e47-9cf28816adbf None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:42.023499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e82cad-5fa5-4b15-8e47-9cf28816adbf None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:42.023643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e82cad-5fa5-4b15-8e47-9cf28816adbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:42.089478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-89e82cad-5fa5-4b15-8e47-9cf28816adbf None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:42.090721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-89e82cad-5fa5-4b15-8e47-9cf28816adbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:42.201805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 754/1472] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:46:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:42.499656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0578c232-578d-48f2-9165-29753b697613 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:42.499934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0578c232-578d-48f2-9165-29753b697613 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:42.500199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0578c232-578d-48f2-9165-29753b697613 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:42.501516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 719/1473] 10.222.0.27 () {54 vars in 962 bytes} [Tue Oct 4 08:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:42.506705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd86fc-50b6-4b01-89c3-b29e5be0598e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:42.506990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd86fc-50b6-4b01-89c3-b29e5be0598e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:42.507264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abd86fc-50b6-4b01-89c3-b29e5be0598e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:42.582997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8abd86fc-50b6-4b01-89c3-b29e5be0598e None None] MFA Rules not processed for user `146238397f514eada174ab3374be5418`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:42.584192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8abd86fc-50b6-4b01-89c3-b29e5be0598e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:42.605091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:42.665861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 755/1474] 10.222.0.27 () {58 vars in 1059 bytes} [Tue Oct 4 08:46:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:42.673554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0578c232-578d-48f2-9165-29753b697613 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:42.675174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0578c232-578d-48f2-9165-29753b697613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:42.780022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0578c232-578d-48f2-9165-29753b697613 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:42.856226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b66a15-1b10-417a-8450-9375939aefc3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:42.856824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b66a15-1b10-417a-8450-9375939aefc3 service glance] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:42.857102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b66a15-1b10-417a-8450-9375939aefc3 service glance] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:42.857371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b66a15-1b10-417a-8450-9375939aefc3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:42.859013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-46b66a15-1b10-417a-8450-9375939aefc3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:42.912270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:42.967573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b66a15-1b10-417a-8450-9375939aefc3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:42.968910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b66a15-1b10-417a-8450-9375939aefc3 service glance] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:42.987517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 720/1475] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:45.237178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33422e47-04bc-44b2-ad37-ffc0feeb798d None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:45.237800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33422e47-04bc-44b2-ad37-ffc0feeb798d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:45.237800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33422e47-04bc-44b2-ad37-ffc0feeb798d None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:45.239070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 756/1476] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:45.245406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3338e94-f6b1-4da9-a4e9-a85915cf5c53 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:45.245815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3338e94-f6b1-4da9-a4e9-a85915cf5c53 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:45.245933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3338e94-f6b1-4da9-a4e9-a85915cf5c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:45.314845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e3338e94-f6b1-4da9-a4e9-a85915cf5c53 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:45.316041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e3338e94-f6b1-4da9-a4e9-a85915cf5c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:45.412807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 721/1477] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:46:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:45.684087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33422e47-04bc-44b2-ad37-ffc0feeb798d None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:45.685542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-33422e47-04bc-44b2-ad37-ffc0feeb798d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:45.822859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33422e47-04bc-44b2-ad37-ffc0feeb798d None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:45.872312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:45.919672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d3edc0-b4a8-49fb-b620-c4e37cf7f3b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:45.920280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d3edc0-b4a8-49fb-b620-c4e37cf7f3b6 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:45.920559 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d3edc0-b4a8-49fb-b620-c4e37cf7f3b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:45.920830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d3edc0-b4a8-49fb-b620-c4e37cf7f3b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:45.921469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d3edc0-b4a8-49fb-b620-c4e37cf7f3b6 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:45.923311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-51d3edc0-b4a8-49fb-b620-c4e37cf7f3b6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:46:45.923311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:46:45.923311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:46:45.924493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d3edc0-b4a8-49fb-b620-c4e37cf7f3b6 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:45.944962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 757/1478] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Oct 4 08:46:45 2022] GET /identity/v3/projects => generated 2553 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:46:48.108636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d367d-f869-4342-b6e6-7a86af74ff8b None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:48.109209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d367d-f869-4342-b6e6-7a86af74ff8b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:48.109209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d367d-f869-4342-b6e6-7a86af74ff8b None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:48.110577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 722/1479] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:46:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:48.115759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f93ad9-629c-4b1e-a76c-44849766a3b7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:48.116192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f93ad9-629c-4b1e-a76c-44849766a3b7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:48.116295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f93ad9-629c-4b1e-a76c-44849766a3b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:48.183204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-03f93ad9-629c-4b1e-a76c-44849766a3b7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:48.184432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-03f93ad9-629c-4b1e-a76c-44849766a3b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:48.292370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 758/1480] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:46:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:48.584931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d367d-f869-4342-b6e6-7a86af74ff8b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:48.586558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-de1d367d-f869-4342-b6e6-7a86af74ff8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:48.596089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:48.732315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d367d-f869-4342-b6e6-7a86af74ff8b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:48.813538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:48.844927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fdf9ae-e3de-4da8-88ee-d77eee8c4243 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:48.845530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fdf9ae-e3de-4da8-88ee-d77eee8c4243 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:48.845807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fdf9ae-e3de-4da8-88ee-d77eee8c4243 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:48.846077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fdf9ae-e3de-4da8-88ee-d77eee8c4243 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:48.847706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-86fdf9ae-e3de-4da8-88ee-d77eee8c4243 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:48.937816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fdf9ae-e3de-4da8-88ee-d77eee8c4243 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:48.939415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fdf9ae-e3de-4da8-88ee-d77eee8c4243 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:48.957117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 723/1481] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:51.280302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d713a1-2e65-4005-ab22-9e07635cc78e None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:51.280874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d713a1-2e65-4005-ab22-9e07635cc78e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:51.280874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d713a1-2e65-4005-ab22-9e07635cc78e None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:51.282159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 759/1482] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:46:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:51.287434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd666985-cfe1-44ef-8255-ad7e2d061e87 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:51.287886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd666985-cfe1-44ef-8255-ad7e2d061e87 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:51.287985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd666985-cfe1-44ef-8255-ad7e2d061e87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:51.354348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-fd666985-cfe1-44ef-8255-ad7e2d061e87 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:51.355569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fd666985-cfe1-44ef-8255-ad7e2d061e87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:51.404401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:51.453617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 724/1483] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:46:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:46:51.742184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d713a1-2e65-4005-ab22-9e07635cc78e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:51.744157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c1d713a1-2e65-4005-ab22-9e07635cc78e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:51.768148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:51.846674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d713a1-2e65-4005-ab22-9e07635cc78e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:51.918736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70261e1f-b1a4-45fb-b707-85fca0312ef2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:51.919336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70261e1f-b1a4-45fb-b707-85fca0312ef2 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:51.919612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70261e1f-b1a4-45fb-b707-85fca0312ef2 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:51.919878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70261e1f-b1a4-45fb-b707-85fca0312ef2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:51.921478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-70261e1f-b1a4-45fb-b707-85fca0312ef2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:51.994537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:52.029672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70261e1f-b1a4-45fb-b707-85fca0312ef2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:52.031145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70261e1f-b1a4-45fb-b707-85fca0312ef2 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:52.055777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 760/1484] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:54.299914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b5eb4f-f8cc-4429-a3af-0ef0692172e2 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:54.300566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b5eb4f-f8cc-4429-a3af-0ef0692172e2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:54.300566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b5eb4f-f8cc-4429-a3af-0ef0692172e2 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:54.301804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 725/1485] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:46:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:54.307354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c89417-1501-466d-8029-810502a5d0bd None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:54.307829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c89417-1501-466d-8029-810502a5d0bd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:54.307938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c89417-1501-466d-8029-810502a5d0bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:54.372539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-92c89417-1501-466d-8029-810502a5d0bd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:54.373767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-92c89417-1501-466d-8029-810502a5d0bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:54.468985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 761/1486] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:46:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:46:54.756633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b5eb4f-f8cc-4429-a3af-0ef0692172e2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:54.758545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-85b5eb4f-f8cc-4429-a3af-0ef0692172e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:54.859974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b5eb4f-f8cc-4429-a3af-0ef0692172e2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:54.932105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:54.933584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d99178-41df-4eb7-91fa-0ae468d14d04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:54.934181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d99178-41df-4eb7-91fa-0ae468d14d04 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:54.934505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d99178-41df-4eb7-91fa-0ae468d14d04 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:54.934773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d99178-41df-4eb7-91fa-0ae468d14d04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:54.936373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-57d99178-41df-4eb7-91fa-0ae468d14d04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:55.044956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d99178-41df-4eb7-91fa-0ae468d14d04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:55.046371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d99178-41df-4eb7-91fa-0ae468d14d04 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:55.064673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 726/1487] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:57.316625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4b7bc6-03df-410c-b850-a4be0efaad50 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:57.317199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4b7bc6-03df-410c-b850-a4be0efaad50 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:57.317199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4b7bc6-03df-410c-b850-a4be0efaad50 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:57.318518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 762/1488] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:46:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:46:57.324432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0886c286-a16e-4533-9677-c948982e5248 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:57.324844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0886c286-a16e-4533-9677-c948982e5248 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:57.324979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0886c286-a16e-4533-9677-c948982e5248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:57.392194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0886c286-a16e-4533-9677-c948982e5248 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:46:57.393400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0886c286-a16e-4533-9677-c948982e5248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:57.444052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:57.486361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 727/1489] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:46:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:46:57.776223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4b7bc6-03df-410c-b850-a4be0efaad50 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:57.777769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1a4b7bc6-03df-410c-b850-a4be0efaad50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:57.813432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:57.878961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4b7bc6-03df-410c-b850-a4be0efaad50 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:57.952256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00aa4ddf-7ade-4519-89a5-7f7fc3cf1eb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:57.952854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00aa4ddf-7ade-4519-89a5-7f7fc3cf1eb2 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:57.953123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00aa4ddf-7ade-4519-89a5-7f7fc3cf1eb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:57.953390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00aa4ddf-7ade-4519-89a5-7f7fc3cf1eb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:57.954996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-00aa4ddf-7ade-4519-89a5-7f7fc3cf1eb2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:46:58.033732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:58.061742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00aa4ddf-7ade-4519-89a5-7f7fc3cf1eb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:58.063322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00aa4ddf-7ade-4519-89a5-7f7fc3cf1eb2 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:58.080839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 763/1490] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:47:00.611819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92443406-ad47-4909-9707-dd32ef474f0e None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:47:00.612378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92443406-ad47-4909-9707-dd32ef474f0e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:47:00.612378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92443406-ad47-4909-9707-dd32ef474f0e None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:47:00.613738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 728/1491] 10.222.0.27 () {54 vars in 927 bytes} [Tue Oct 4 08:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:47:00.619495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ad57bf-37f7-4a21-be46-a4954bcac467 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:47:00.619932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ad57bf-37f7-4a21-be46-a4954bcac467 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:47:00.620025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ad57bf-37f7-4a21-be46-a4954bcac467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:47:00.803052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-46ad57bf-37f7-4a21-be46-a4954bcac467 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:47:00.804268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-46ad57bf-37f7-4a21-be46-a4954bcac467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:47:00.897140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 764/1492] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Oct 4 08:47:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:47:01.185417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92443406-ad47-4909-9707-dd32ef474f0e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:47:01.186990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-92443406-ad47-4909-9707-dd32ef474f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:47:01.288082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92443406-ad47-4909-9707-dd32ef474f0e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:47:01.361649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fced9cc-4a2b-40dd-99d4-75d27dea2635 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:47:01.362246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fced9cc-4a2b-40dd-99d4-75d27dea2635 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:47:01.362547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fced9cc-4a2b-40dd-99d4-75d27dea2635 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:47:01.362814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fced9cc-4a2b-40dd-99d4-75d27dea2635 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:47:01.364388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4fced9cc-4a2b-40dd-99d4-75d27dea2635 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:47:01.373393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:47:01.476993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fced9cc-4a2b-40dd-99d4-75d27dea2635 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:47:01.478841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fced9cc-4a2b-40dd-99d4-75d27dea2635 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:47:01.496799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 729/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:47:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:47:10.895646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=82 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:47:11.494541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=50 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:23.950810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:23.951420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:23.951420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:24.032118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:48:24.033323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:24.124695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 765/1494] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:48:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:48:24.133507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0462f17-8cbf-45ff-8d9d-014ec9b4d86e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:24.134042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0462f17-8cbf-45ff-8d9d-014ec9b4d86e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:24.134042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0462f17-8cbf-45ff-8d9d-014ec9b4d86e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:24.209481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c0462f17-8cbf-45ff-8d9d-014ec9b4d86e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:48:24.210651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c0462f17-8cbf-45ff-8d9d-014ec9b4d86e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:24.305465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 730/1495] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:48:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:48:24.314139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:24.315425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:24.327397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67028930-9109-4f1a-b790-d6c2c7a82876 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:24.389984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:24.422971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:24.423573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:24.423843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:24.424111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:24.424790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:48:24.425434 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:24.426981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:24.426981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:48:24.426981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:48:24.427862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:24.439451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 766/1496] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:48:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:48:24.448360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0462f17-8cbf-45ff-8d9d-014ec9b4d86e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:24.457563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0462f17-8cbf-45ff-8d9d-014ec9b4d86e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:24.519972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:24.552567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:24.553188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:24.553465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:24.553739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:24.555100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:24.556825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING py.warnings [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:24.556825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: warnings.warn(msg) Oct 04 08:48:24.556825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: Oct 04 08:48:24.558016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:24.668406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 731/1497] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:48:24 2022] POST /identity/v3/projects => generated 377 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:48:24.672761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:24.674335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:24.708377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35a4a16-3ea9-46aa-b8cd-0b3342fd092e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:24.798651 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:24.799172 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:24.799447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:24.799707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:24.800311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:24.801298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:24.801298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:48:24.801298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:48:24.802506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:24.922558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 767/1498] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:48:24 2022] POST /identity/v3/users => generated 289 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:48:24.926753 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:24.928356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:24.992122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117e2b2-08b5-43e2-9c22-855e67d79aa3 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:25.075181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:25.084925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:25.086208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:25.086511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:25.086780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:25.087413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:25.088401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING py.warnings [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:25.088401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: warnings.warn(msg) Oct 04 08:48:25.088401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: Oct 04 08:48:25.089281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:25.103051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 732/1499] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:48:24 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:48:25.106988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:25.108554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:25.119913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0120f388-bc38-4057-af2b-4c13e498d300 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:25.144892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:25.213908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e1c9f8-70d1-46e4-bd0c-065fde30945c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:25.214544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e1c9f8-70d1-46e4-bd0c-065fde30945c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:25.214816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e1c9f8-70d1-46e4-bd0c-065fde30945c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:25.215080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e1c9f8-70d1-46e4-bd0c-065fde30945c admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:25.215711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e1c9f8-70d1-46e4-bd0c-065fde30945c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:25.216718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-84e1c9f8-70d1-46e4-bd0c-065fde30945c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:25.216718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:48:25.216718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:48:25.217594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e1c9f8-70d1-46e4-bd0c-065fde30945c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:25.230372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 768/1500] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:48:25 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:48:25.234449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:25.236009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:25.269383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f113f8b2-ee36-46af-8a37-23a44b1c11a0 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:25.363369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597abfbf-e4b5-40a2-9f33-649ad797f943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:25.363937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597abfbf-e4b5-40a2-9f33-649ad797f943 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:25.364210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597abfbf-e4b5-40a2-9f33-649ad797f943 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:25.364498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597abfbf-e4b5-40a2-9f33-649ad797f943 admin admin] PATH_INFO: `/v3/projects/fda18ddd041046178357b5c549154169/users/dd171e00679644f38f0cc06d2f1d5e3a/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:25.387610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597abfbf-e4b5-40a2-9f33-649ad797f943 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fda18ddd041046178357b5c549154169, user_id=dd171e00679644f38f0cc06d2f1d5e3a, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:25.388631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING py.warnings [None req-597abfbf-e4b5-40a2-9f33-649ad797f943 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:25.388631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: warnings.warn(msg) Oct 04 08:48:25.388631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: Oct 04 08:48:25.390588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597abfbf-e4b5-40a2-9f33-649ad797f943 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:25.471671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 733/1501] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:48:25 2022] PUT /identity/v3/projects/fda18ddd041046178357b5c549154169/users/dd171e00679644f38f0cc06d2f1d5e3a/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:48:25.483700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320a085-48e3-4a63-8c54-31611702a137 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:25.483985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320a085-48e3-4a63-8c54-31611702a137 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:25.487142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320a085-48e3-4a63-8c54-31611702a137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:25.539840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7320a085-48e3-4a63-8c54-31611702a137 None None] MFA Rules not processed for user `dd171e00679644f38f0cc06d2f1d5e3a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:48:25.541035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7320a085-48e3-4a63-8c54-31611702a137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:25.614791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 769/1502] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:48:25 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:48:25.627799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e2e11-96d3-4700-bb53-fd73f1b970ec None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:25.628081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e2e11-96d3-4700-bb53-fd73f1b970ec None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:25.628350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e2e11-96d3-4700-bb53-fd73f1b970ec None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:25.629949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 734/1503] 10.222.0.27 () {54 vars in 963 bytes} [Tue Oct 4 08:48:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:48:25.635306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d58a36-1af9-4e3c-bf98-9c65029d636a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:25.635589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d58a36-1af9-4e3c-bf98-9c65029d636a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:25.635857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d58a36-1af9-4e3c-bf98-9c65029d636a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:25.666708 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:25.711566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-67d58a36-1af9-4e3c-bf98-9c65029d636a None None] MFA Rules not processed for user `f49a19f9484d4aa99567e66b4fc396ef`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:48:25.712763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-67d58a36-1af9-4e3c-bf98-9c65029d636a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:25.786754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 770/1504] 10.222.0.27 () {58 vars in 1060 bytes} [Tue Oct 4 08:48:25 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:48:25.796643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e2e11-96d3-4700-bb53-fd73f1b970ec None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:25.798181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-139e2e11-96d3-4700-bb53-fd73f1b970ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:25.848091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:25.902979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e2e11-96d3-4700-bb53-fd73f1b970ec None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:25.975746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:25.976343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:25.976619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:25.976883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:25.978495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:26.077080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:26.078528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:26.081956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:26.096150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 735/1505] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:48:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:48:26.143610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3ef51b-ac55-4895-a9a4-4c4d41175da7 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:26.144049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3ef51b-ac55-4895-a9a4-4c4d41175da7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:26.144144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3ef51b-ac55-4895-a9a4-4c4d41175da7 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:26.145544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 771/1506] 10.222.0.27 () {52 vars in 830 bytes} [Tue Oct 4 08:48:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:48:26.149327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:26.150590 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:26.250550 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5fdf81-f2cb-4316-8c45-232475a0a360 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:26.341432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:26.342053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:26.342327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:26.342642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] PATH_INFO: `/v3/users/dd171e00679644f38f0cc06d2f1d5e3a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:26.355603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd171e00679644f38f0cc06d2f1d5e3a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:26.356609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING py.warnings [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:26.356609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: warnings.warn(msg) Oct 04 08:48:26.356609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: Oct 04 08:48:26.357820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:26.367876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:26.605481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd171e00679644f38f0cc06d2f1d5e3a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:26.619074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd171e00679644f38f0cc06d2f1d5e3a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:26.619523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd171e00679644f38f0cc06d2f1d5e3a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:26.620043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd171e00679644f38f0cc06d2f1d5e3a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:26.674933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd171e00679644f38f0cc06d2f1d5e3a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:26.703829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 736/1507] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:48:26 2022] DELETE /identity/v3/users/dd171e00679644f38f0cc06d2f1d5e3a => generated 0 bytes in 555 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:48:26.708430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:26.708741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:26.709016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:26.774647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:48:26.791851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:26.851911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:26.873936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 772/1508] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:48:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:48:26.884444 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:26.890608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:26.986369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ed7302-fadc-4815-b442-d752e0dc62d2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:27.088493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:27.107431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e3a27d-69ef-4674-9fda-fd998aba44b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:27.108043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e3a27d-69ef-4674-9fda-fd998aba44b3 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:27.108320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e3a27d-69ef-4674-9fda-fd998aba44b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:27.108590 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e3a27d-69ef-4674-9fda-fd998aba44b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:27.110187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-76e3a27d-69ef-4674-9fda-fd998aba44b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:27.221422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e3a27d-69ef-4674-9fda-fd998aba44b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:27.222810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e3a27d-69ef-4674-9fda-fd998aba44b3 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:27.241135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 737/1509] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:48:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:48:27.752797 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:27.754295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:48:27.766110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc1f851-b517-4e3b-9ede-1fbf76f06cbe None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:27.860984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:27.861643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:27.861918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:27.862188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] PATH_INFO: `/v3/projects/fda18ddd041046178357b5c549154169` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:27.872570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] RBAC: Authorizing `identity:delete_project(project_id=fda18ddd041046178357b5c549154169)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:27.873588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:48:27.873588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:48:27.873588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:48:27.874821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:28.028839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fda18ddd041046178357b5c549154169', 'request_id': 'req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:28.042447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fda18ddd041046178357b5c549154169', 'request_id': 'req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:28.114484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fda18ddd041046178357b5c549154169', 'request_id': 'req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:28.139015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] The token cache is being invalidate because project fda18ddd041046178357b5c549154169 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:48:28.139301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:28.139723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87d6a813-f3ce-47aa-ac9c-c521b14fabfb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:48:28.141096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 773/1510] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:48:27 2022] DELETE /identity/v3/projects/fda18ddd041046178357b5c549154169 => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:48:37.238989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=62 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:38.137109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:49:54.942695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f77b5c3-e939-45e7-9465-434d7dbb1788 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:54.943320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f77b5c3-e939-45e7-9465-434d7dbb1788 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:54.943320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f77b5c3-e939-45e7-9465-434d7dbb1788 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:54.944584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 738/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:49:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:49:54.950111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f0eb9e-7ba7-48c9-94b7-01f8c963c31b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:54.950583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f0eb9e-7ba7-48c9-94b7-01f8c963c31b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:54.950688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f0eb9e-7ba7-48c9-94b7-01f8c963c31b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:55.030946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-27f0eb9e-7ba7-48c9-94b7-01f8c963c31b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:49:55.032137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-27f0eb9e-7ba7-48c9-94b7-01f8c963c31b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:49:55.137375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 774/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:49:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:49:55.298071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f77b5c3-e939-45e7-9465-434d7dbb1788 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:49:55.300259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5f77b5c3-e939-45e7-9465-434d7dbb1788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:49:55.418568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f77b5c3-e939-45e7-9465-434d7dbb1788 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:49:55.423914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb93ea9-aec6-49e6-b514-c0e066c6922f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:49:55.424511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb93ea9-aec6-49e6-b514-c0e066c6922f service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:55.424786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb93ea9-aec6-49e6-b514-c0e066c6922f service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:55.425051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb93ea9-aec6-49e6-b514-c0e066c6922f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:55.433608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb93ea9-aec6-49e6-b514-c0e066c6922f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:49:55.435232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb93ea9-aec6-49e6-b514-c0e066c6922f service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:49:55.444842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 739/1513] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:49:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:49:56.585550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce5108-5440-4465-bfc3-3e3b7e2b1eba None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:56.586710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce5108-5440-4465-bfc3-3e3b7e2b1eba None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:56.586710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce5108-5440-4465-bfc3-3e3b7e2b1eba None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:56.588254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 775/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:49:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:49:56.594587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021fff2b-5d57-41ec-8c4f-2db3f697a2e9 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:56.595087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021fff2b-5d57-41ec-8c4f-2db3f697a2e9 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:56.595206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021fff2b-5d57-41ec-8c4f-2db3f697a2e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:56.614336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-021fff2b-5d57-41ec-8c4f-2db3f697a2e9 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:49:56.615554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-021fff2b-5d57-41ec-8c4f-2db3f697a2e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:49:56.623597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 740/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:49:56.782146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ce5108-5440-4465-bfc3-3e3b7e2b1eba None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:49:56.796280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ce5108-5440-4465-bfc3-3e3b7e2b1eba None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:49:56.801798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3022958a-f410-4115-ab75-fe71a81eb77b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:49:56.802390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022958a-f410-4115-ab75-fe71a81eb77b service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:56.802688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022958a-f410-4115-ab75-fe71a81eb77b service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:56.802954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022958a-f410-4115-ab75-fe71a81eb77b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:56.811366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3022958a-f410-4115-ab75-fe71a81eb77b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:49:56.812939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3022958a-f410-4115-ab75-fe71a81eb77b service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:49:56.824928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 776/1516] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:49:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:49:56.891376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656ea81f-af15-4959-961c-9d8a27f18e54 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:56.891660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656ea81f-af15-4959-961c-9d8a27f18e54 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:56.891937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656ea81f-af15-4959-961c-9d8a27f18e54 None None] PATH_INFO: `/v3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:56.893307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 741/1517] 10.222.0.42 () {54 vars in 925 bytes} [Tue Oct 4 08:49:56 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:49:56.900241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc0afdd-802d-440e-b4c5-22583ec6e545 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:49:56.900569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc0afdd-802d-440e-b4c5-22583ec6e545 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:49:56.900818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc0afdd-802d-440e-b4c5-22583ec6e545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:49:56.965478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-efc0afdd-802d-440e-b4c5-22583ec6e545 None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:49:56.966731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-efc0afdd-802d-440e-b4c5-22583ec6e545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:49:56.983837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:49:57.043747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 777/1518] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Oct 4 08:49:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:00.923222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec047378-0dee-47dd-8e8d-0405c6e13459 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:00.923688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec047378-0dee-47dd-8e8d-0405c6e13459 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:00.923796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec047378-0dee-47dd-8e8d-0405c6e13459 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:00.925075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 742/1519] 10.222.0.27 () {54 vars in 961 bytes} [Tue Oct 4 08:50:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:50:00.930628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f73af9-6e42-473c-8b4a-b2e8415ad095 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:00.931112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f73af9-6e42-473c-8b4a-b2e8415ad095 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:00.931232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f73af9-6e42-473c-8b4a-b2e8415ad095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:00.951553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c9f73af9-6e42-473c-8b4a-b2e8415ad095 None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:50:00.953119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c9f73af9-6e42-473c-8b4a-b2e8415ad095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:00.961587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 778/1520] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Oct 4 08:50:00 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:00.969225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec047378-0dee-47dd-8e8d-0405c6e13459 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:50:00.982806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec047378-0dee-47dd-8e8d-0405c6e13459 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:50:00.987606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6815fa4c-b0eb-4f0d-a6bf-7dbece06609d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:50:00.988190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6815fa4c-b0eb-4f0d-a6bf-7dbece06609d service placement] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:00.988461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6815fa4c-b0eb-4f0d-a6bf-7dbece06609d service placement] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:00.988729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6815fa4c-b0eb-4f0d-a6bf-7dbece06609d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:00.997320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6815fa4c-b0eb-4f0d-a6bf-7dbece06609d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:50:00.998841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6815fa4c-b0eb-4f0d-a6bf-7dbece06609d service placement] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:50:01.008374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 743/1521] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:50:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:50:08.334208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bc0bf9-eebc-4501-ae72-9f9290da8af6 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:08.334800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bc0bf9-eebc-4501-ae72-9f9290da8af6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:08.334800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bc0bf9-eebc-4501-ae72-9f9290da8af6 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:08.336687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 779/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:50:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:50:08.343038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583ba36a-7c2c-449b-8712-39bee2cc4588 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:08.343462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583ba36a-7c2c-449b-8712-39bee2cc4588 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:08.343573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583ba36a-7c2c-449b-8712-39bee2cc4588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:08.363433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-583ba36a-7c2c-449b-8712-39bee2cc4588 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:50:08.364647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-583ba36a-7c2c-449b-8712-39bee2cc4588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:08.375736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 744/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:50:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:08.527383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bc0bf9-eebc-4501-ae72-9f9290da8af6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:50:08.542241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bc0bf9-eebc-4501-ae72-9f9290da8af6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:50:08.547975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd46a2e6-e109-46e8-a289-e130d6f1a5ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:50:08.548567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd46a2e6-e109-46e8-a289-e130d6f1a5ac service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:08.548842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd46a2e6-e109-46e8-a289-e130d6f1a5ac service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:08.549108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd46a2e6-e109-46e8-a289-e130d6f1a5ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:08.557850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd46a2e6-e109-46e8-a289-e130d6f1a5ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:50:08.559453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd46a2e6-e109-46e8-a289-e130d6f1a5ac service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:50:08.568790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 780/1524] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:50:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:50:09.671438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3452c702-ab7d-45fc-8d30-c862b0ddee19 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:09.671978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3452c702-ab7d-45fc-8d30-c862b0ddee19 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:09.671978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3452c702-ab7d-45fc-8d30-c862b0ddee19 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:09.673252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 745/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:50:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:50:09.679421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8446b60-f0d2-4f1c-b8ba-fa78c71e7f59 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:09.679844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8446b60-f0d2-4f1c-b8ba-fa78c71e7f59 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:09.680603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8446b60-f0d2-4f1c-b8ba-fa78c71e7f59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:09.701141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b8446b60-f0d2-4f1c-b8ba-fa78c71e7f59 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:50:09.702365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b8446b60-f0d2-4f1c-b8ba-fa78c71e7f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:09.710358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 781/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:50:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:09.874564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3452c702-ab7d-45fc-8d30-c862b0ddee19 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:50:09.889503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3452c702-ab7d-45fc-8d30-c862b0ddee19 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:50:09.894799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9082c7e-4535-4ee7-be0a-fec8526b6a27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:50:09.895392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9082c7e-4535-4ee7-be0a-fec8526b6a27 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:09.895663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9082c7e-4535-4ee7-be0a-fec8526b6a27 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:09.895929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9082c7e-4535-4ee7-be0a-fec8526b6a27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:09.904147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9082c7e-4535-4ee7-be0a-fec8526b6a27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:50:09.905707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9082c7e-4535-4ee7-be0a-fec8526b6a27 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:50:09.915179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 746/1527] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:50:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:50:14.243276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e037b210-c70e-47ca-a665-cba93f80c379 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:14.243845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e037b210-c70e-47ca-a665-cba93f80c379 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:14.243845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e037b210-c70e-47ca-a665-cba93f80c379 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:14.245940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 782/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:50:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:50:14.251312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33575196-47c1-48bb-8012-4b0ad4f5664d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:14.251737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33575196-47c1-48bb-8012-4b0ad4f5664d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:14.251858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33575196-47c1-48bb-8012-4b0ad4f5664d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:14.271037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-33575196-47c1-48bb-8012-4b0ad4f5664d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:50:14.272217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-33575196-47c1-48bb-8012-4b0ad4f5664d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:14.280975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 747/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:14.573340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e037b210-c70e-47ca-a665-cba93f80c379 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:50:14.575075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e037b210-c70e-47ca-a665-cba93f80c379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:14.654300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e037b210-c70e-47ca-a665-cba93f80c379 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:50:14.657897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0a0148-4091-4132-81b0-524c62207212 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:50:14.658524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a0148-4091-4132-81b0-524c62207212 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:14.658802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a0148-4091-4132-81b0-524c62207212 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:14.659074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a0148-4091-4132-81b0-524c62207212 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:14.668030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0a0148-4091-4132-81b0-524c62207212 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:50:14.669450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0a0148-4091-4132-81b0-524c62207212 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:50:14.678658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 783/1530] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:50:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:50:16.840247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b0d0bf-4621-45ea-832a-72e14c4013a8 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:16.840826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b0d0bf-4621-45ea-832a-72e14c4013a8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:16.840826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b0d0bf-4621-45ea-832a-72e14c4013a8 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:16.842157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 748/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:50:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:50:16.847571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611a133f-7595-416d-bc00-43d1e9d7a5e6 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:16.847959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611a133f-7595-416d-bc00-43d1e9d7a5e6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:16.852811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611a133f-7595-416d-bc00-43d1e9d7a5e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:16.868591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-611a133f-7595-416d-bc00-43d1e9d7a5e6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:50:16.869771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-611a133f-7595-416d-bc00-43d1e9d7a5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:16.877887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 784/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:50:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:17.166962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b0d0bf-4621-45ea-832a-72e14c4013a8 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:50:17.177609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b0d0bf-4621-45ea-832a-72e14c4013a8 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:50:17.181934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b545b5b-bfaa-4612-80ab-53ca95d707a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:50:17.182562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b545b5b-bfaa-4612-80ab-53ca95d707a4 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:17.182836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b545b5b-bfaa-4612-80ab-53ca95d707a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:17.183102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b545b5b-bfaa-4612-80ab-53ca95d707a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:17.191686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b545b5b-bfaa-4612-80ab-53ca95d707a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:50:17.193083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b545b5b-bfaa-4612-80ab-53ca95d707a4 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:50:17.202255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 749/1533] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:50:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:50:21.316666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3831b4-09a5-447c-8577-ab41b6eeb62f None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:21.317233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3831b4-09a5-447c-8577-ab41b6eeb62f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:21.317233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3831b4-09a5-447c-8577-ab41b6eeb62f None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:21.318984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 785/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:50:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:50:21.324843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2857cb-8e35-4dfd-9237-908599fb4745 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:21.325283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2857cb-8e35-4dfd-9237-908599fb4745 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:21.325406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2857cb-8e35-4dfd-9237-908599fb4745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:21.332938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=81 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:50:21.344996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-bf2857cb-8e35-4dfd-9237-908599fb4745 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:50:21.346751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bf2857cb-8e35-4dfd-9237-908599fb4745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:21.354955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 750/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:50:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:21.667599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3831b4-09a5-447c-8577-ab41b6eeb62f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:50:21.669290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3e3831b4-09a5-447c-8577-ab41b6eeb62f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:21.733097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:50:21.746657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3831b4-09a5-447c-8577-ab41b6eeb62f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:50:21.750262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c05983a-0773-435f-9ba1-fca3f3d25edc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:50:21.750890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c05983a-0773-435f-9ba1-fca3f3d25edc service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:21.751169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c05983a-0773-435f-9ba1-fca3f3d25edc service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:21.751437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c05983a-0773-435f-9ba1-fca3f3d25edc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:21.761832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c05983a-0773-435f-9ba1-fca3f3d25edc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:50:21.763306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c05983a-0773-435f-9ba1-fca3f3d25edc service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:50:21.773079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 786/1536] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:50:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:50:23.931012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135dbfa8-78cb-47ae-956e-9c26ffe35327 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:23.931554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135dbfa8-78cb-47ae-956e-9c26ffe35327 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:23.931554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135dbfa8-78cb-47ae-956e-9c26ffe35327 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:23.932883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 751/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:50:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:50:23.938104 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be0672-e251-44ae-85d4-71230295ce51 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:23.938623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be0672-e251-44ae-85d4-71230295ce51 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:23.938723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be0672-e251-44ae-85d4-71230295ce51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:23.958737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d0be0672-e251-44ae-85d4-71230295ce51 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:50:23.959951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d0be0672-e251-44ae-85d4-71230295ce51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:23.968606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 787/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:50:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:50:24.275243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135dbfa8-78cb-47ae-956e-9c26ffe35327 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:50:24.276932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-135dbfa8-78cb-47ae-956e-9c26ffe35327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:50:24.290427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135dbfa8-78cb-47ae-956e-9c26ffe35327 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:50:24.293970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb518b86-b6ff-462f-a6cc-f3505c356df0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:50:24.294601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb518b86-b6ff-462f-a6cc-f3505c356df0 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:50:24.294883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb518b86-b6ff-462f-a6cc-f3505c356df0 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:50:24.295155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb518b86-b6ff-462f-a6cc-f3505c356df0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:50:24.303912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb518b86-b6ff-462f-a6cc-f3505c356df0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:50:24.305436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb518b86-b6ff-462f-a6cc-f3505c356df0 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:50:24.314749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 752/1539] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:50:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:50:33.953681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=11 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:50:34.310606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=8 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:49.874377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cf5a3-5470-4b14-a90f-33bfd3996092 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:49.875820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cf5a3-5470-4b14-a90f-33bfd3996092 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:49.876405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cf5a3-5470-4b14-a90f-33bfd3996092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:49.907267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f86cf5a3-5470-4b14-a90f-33bfd3996092 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:49.908848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f86cf5a3-5470-4b14-a90f-33bfd3996092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:49.920834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 788/1540] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:49.930720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806b45-3aa0-481d-9253-63458221a072 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:49.931277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806b45-3aa0-481d-9253-63458221a072 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:49.931277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a806b45-3aa0-481d-9253-63458221a072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:49.962568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-2a806b45-3aa0-481d-9253-63458221a072 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:49.962568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2a806b45-3aa0-481d-9253-63458221a072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:49.974295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 753/1541] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:49.979799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cf5a3-5470-4b14-a90f-33bfd3996092 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:49.993990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d1ba03-9569-4e02-b951-5f9f2dd57884 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:49.994396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d1ba03-9569-4e02-b951-5f9f2dd57884 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:49.994612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d1ba03-9569-4e02-b951-5f9f2dd57884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.000485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cf5a3-5470-4b14-a90f-33bfd3996092 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.012889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f29301-302a-4dff-9730-4914e869cac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.014228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f29301-302a-4dff-9730-4914e869cac3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.014795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f29301-302a-4dff-9730-4914e869cac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.015318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f29301-302a-4dff-9730-4914e869cac3 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.016361 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f29301-302a-4dff-9730-4914e869cac3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.016939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f29301-302a-4dff-9730-4914e869cac3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.021364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f29301-302a-4dff-9730-4914e869cac3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.029593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a2d1ba03-9569-4e02-b951-5f9f2dd57884 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.030989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a2d1ba03-9569-4e02-b951-5f9f2dd57884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.045880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 789/1542] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.047702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf605aa8-ca84-4948-9143-e643f6e871ae None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.047916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 754/1543] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:51:50.049025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf605aa8-ca84-4948-9143-e643f6e871ae None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.049545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf605aa8-ca84-4948-9143-e643f6e871ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.061588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d1ba03-9569-4e02-b951-5f9f2dd57884 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.079370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d1ba03-9569-4e02-b951-5f9f2dd57884 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.082599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-bf605aa8-ca84-4948-9143-e643f6e871ae None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.084370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-bf605aa8-ca84-4948-9143-e643f6e871ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.092086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.092086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.092086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.092086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.092510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.093009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.113020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 790/1544] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.115080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5180a46-388b-4893-a653-2d70744c56cc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.116718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5180a46-388b-4893-a653-2d70744c56cc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.117292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5180a46-388b-4893-a653-2d70744c56cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.151716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f5180a46-388b-4893-a653-2d70744c56cc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.152852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f5180a46-388b-4893-a653-2d70744c56cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.161487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 791/1545] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.163893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63bae15-69cc-4de9-aafb-c57b003ab7e8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.163893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63bae15-69cc-4de9-aafb-c57b003ab7e8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.163893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63bae15-69cc-4de9-aafb-c57b003ab7e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.186165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b63bae15-69cc-4de9-aafb-c57b003ab7e8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.187484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b63bae15-69cc-4de9-aafb-c57b003ab7e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.199898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 792/1546] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.201214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918bbe6-6845-4258-863f-ab5333cf7cb0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.201500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918bbe6-6845-4258-863f-ab5333cf7cb0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.201773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918bbe6-6845-4258-863f-ab5333cf7cb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.221992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7918bbe6-6845-4258-863f-ab5333cf7cb0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.223217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7918bbe6-6845-4258-863f-ab5333cf7cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.231771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 793/1547] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.233523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fef60-c233-43c6-a125-004449ce329d None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.233523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fef60-c233-43c6-a125-004449ce329d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.233523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fef60-c233-43c6-a125-004449ce329d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.253049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-666fef60-c233-43c6-a125-004449ce329d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.254260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-666fef60-c233-43c6-a125-004449ce329d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.263349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 794/1548] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.266922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 755/1549] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/projects => generated 387 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:51:50.267101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8c1512-583d-4649-9b87-e2a52e618edc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.267209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.267313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8c1512-583d-4649-9b87-e2a52e618edc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.267313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8c1512-583d-4649-9b87-e2a52e618edc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.288685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-cc8c1512-583d-4649-9b87-e2a52e618edc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.296541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cc8c1512-583d-4649-9b87-e2a52e618edc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.336592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de5bc6-ed27-4ff4-9538-26c5bb87e76d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.342205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.342830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.343103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.343370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.344007 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.344303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.345440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING py.warnings [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:51:50.345440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: warnings.warn(msg) Oct 04 08:51:50.345440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: Oct 04 08:51:50.346319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.354738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 795/1550] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.359399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42172c78-0ff3-4463-9afd-fc2c322d0c83 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.359399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42172c78-0ff3-4463-9afd-fc2c322d0c83 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.359399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42172c78-0ff3-4463-9afd-fc2c322d0c83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.377784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-42172c78-0ff3-4463-9afd-fc2c322d0c83 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.379072 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-42172c78-0ff3-4463-9afd-fc2c322d0c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.387924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 796/1551] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.389189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ede0e9-1628-408e-ade4-eb77798ddaf7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.389465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ede0e9-1628-408e-ade4-eb77798ddaf7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.389729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ede0e9-1628-408e-ade4-eb77798ddaf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.409090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-96ede0e9-1628-408e-ade4-eb77798ddaf7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.410183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-96ede0e9-1628-408e-ade4-eb77798ddaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.418258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 797/1552] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.426667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c164d-1a3c-4669-a147-2fcba996b039 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.426667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c164d-1a3c-4669-a147-2fcba996b039 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.426667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c164d-1a3c-4669-a147-2fcba996b039 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.439629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-386c164d-1a3c-4669-a147-2fcba996b039 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.440816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-386c164d-1a3c-4669-a147-2fcba996b039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.449573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 756/1553] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.449573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.452470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 798/1554] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.454010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386c164d-1a3c-4669-a147-2fcba996b039 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.458918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f9dc6-b991-482b-806b-9544d78e4416 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.464685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386c164d-1a3c-4669-a147-2fcba996b039 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.466290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33f7a7d-e58e-4c77-9572-6c4d3a166b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.466290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33f7a7d-e58e-4c77-9572-6c4d3a166b0d admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.466290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33f7a7d-e58e-4c77-9572-6c4d3a166b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.466290 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33f7a7d-e58e-4c77-9572-6c4d3a166b0d admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.466654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33f7a7d-e58e-4c77-9572-6c4d3a166b0d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.466984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33f7a7d-e58e-4c77-9572-6c4d3a166b0d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.468562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33f7a7d-e58e-4c77-9572-6c4d3a166b0d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.469606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.470288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.470598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.470870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.471495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.473399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.481407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 757/1555] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.482963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd273cd-cb3d-4a3d-9b00-5a9098516e5f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd273cd-cb3d-4a3d-9b00-5a9098516e5f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.484199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd273cd-cb3d-4a3d-9b00-5a9098516e5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.504291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9dd273cd-cb3d-4a3d-9b00-5a9098516e5f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.505774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9dd273cd-cb3d-4a3d-9b00-5a9098516e5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.514486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 758/1556] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.515657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad7503-ddf6-4aa5-81e5-5144980c7eab None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.515974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad7503-ddf6-4aa5-81e5-5144980c7eab None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.516250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad7503-ddf6-4aa5-81e5-5144980c7eab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.535419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e7ad7503-ddf6-4aa5-81e5-5144980c7eab None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.536595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e7ad7503-ddf6-4aa5-81e5-5144980c7eab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.544631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 759/1557] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.545904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9315101-9203-443d-beeb-97a9c75835e8 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.546183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9315101-9203-443d-beeb-97a9c75835e8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.546479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9315101-9203-443d-beeb-97a9c75835e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.563796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 799/1558] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/users => generated 294 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:50.564765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.566164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b9315101-9203-443d-beeb-97a9c75835e8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.567421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b9315101-9203-443d-beeb-97a9c75835e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.575906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc74e17-0435-401e-b9aa-c4a26546fb7c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.576645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 760/1559] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.577421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9315101-9203-443d-beeb-97a9c75835e8 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.580975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a812f8de-b51d-43b8-9775-d9d7b2cc2f88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.581579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812f8de-b51d-43b8-9775-d9d7b2cc2f88 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.581854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812f8de-b51d-43b8-9775-d9d7b2cc2f88 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.582128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812f8de-b51d-43b8-9775-d9d7b2cc2f88 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.582796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a812f8de-b51d-43b8-9775-d9d7b2cc2f88 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.583093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a812f8de-b51d-43b8-9775-d9d7b2cc2f88 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.585051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a812f8de-b51d-43b8-9775-d9d7b2cc2f88 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.587165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9315101-9203-443d-beeb-97a9c75835e8 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.591944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.593239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.593526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.593797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.594427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.596372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.597725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 800/1560] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.599019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c73314-faac-4d5f-b574-997e04ae6eea None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.599296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c73314-faac-4d5f-b574-997e04ae6eea None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.599561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c73314-faac-4d5f-b574-997e04ae6eea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.619829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f4c73314-faac-4d5f-b574-997e04ae6eea None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:50.621031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f4c73314-faac-4d5f-b574-997e04ae6eea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:50.630299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 801/1561] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:50.630299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c73314-faac-4d5f-b574-997e04ae6eea None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.640031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c73314-faac-4d5f-b574-997e04ae6eea None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.644595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.645167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.645438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.645720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.646313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.647376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:51:50.647376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:51:50.647376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:51:50.648588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.685551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 761/1562] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/projects => generated 403 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:50.686465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.743833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:50.748808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4b922b-fd04-4fdf-861f-d47ff7e9538c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.753486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.754121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.754421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.754727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.755612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.755916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.757599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.765278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 802/1563] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/projects => generated 393 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:50.766184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.769603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 762/1564] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.770492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.780127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe289ad-2fd6-4e74-ac36-42835534cef8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.805668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:50.836707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b881d8f-bd20-4798-9a8e-62bf9a3b07bc admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.840174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.840863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.841145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.841413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.842049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.842327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.845527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.845527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.845527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.845804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.845897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.845897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.846725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.854068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 763/1565] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:50.854918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.860521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 803/1566] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.861830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.872022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa75bb74-7a2a-46d0-905c-8866a3e7a89d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.872199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598e9e19-e352-4533-83b4-2c0be955bad1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.876604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.877086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.877204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.877306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.877544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.877661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.877819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.877923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.878473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.878584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.878777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.880606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.880726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.892980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 764/1567] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.895494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.905592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a89fa1-04de-4584-9fc4-3c27fed7a48e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.910193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.910810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.911086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.911356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.912016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:51:50.912874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.914831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.926448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 765/1568] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:51:50.927201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:50.937595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27530495-e9e6-494b-9f08-660ed79c2f0a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:50.942177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:50.942787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:50.943064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:50.943334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:50.944057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:50.945085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING py.warnings [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:51:50.945085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: warnings.warn(msg) Oct 04 08:51:50.945085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: Oct 04 08:51:50.946284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:50.954814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 804/1569] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/projects => generated 383 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:50.954814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.015119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e7a57-d077-4392-8cd3-ffb2de8a4e4a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.019754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.020362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.020644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.020919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.021544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.023530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.044440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 766/1570] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/users => generated 302 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.047754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.055804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04810c9-d994-40a1-9b5a-c8920147ef81 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.060515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.061120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.061396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.061662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.062282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.064250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.123958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 805/1571] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:50 2022] POST /identity/v3/users => generated 297 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.124793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.134670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb46826-bd32-4aaf-8f14-654c08152bf9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.139233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.139809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.140092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.140353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.140981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.142607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.148507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 767/1572] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/projects => generated 379 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.150899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.153467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 806/1573] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:51.154115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.164820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a47a00e-02bb-43a8-9a83-d8894008f514 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.216224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a50ffd6-24ca-472a-8f9e-2ad5ef81c135 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.220755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.221359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.221637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.221907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.222566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.223049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.223724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.224074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.224412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.224531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.225175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.228103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.340431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 768/1574] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/projects => generated 369 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.340431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.343663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:51.352192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 807/1575] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/projects => generated 373 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.352192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.354909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:51.409852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6bc476-c119-470e-82a5-5380cdbd5515 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.412709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6106928a-385d-41ab-abad-c668b4a2a05e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.416382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.419200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.419200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.419200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.419472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.419584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.419693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.419693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.419693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.419693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.420075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.421439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.529450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 808/1576] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/users => generated 292 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.532642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.539671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5292e5-efbb-4017-9f64-8872553281d8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.544319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.544900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.545179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.545449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.546074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.547692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.557972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 809/1577] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:51.558603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.565519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 769/1578] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/projects => generated 373 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.565519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.571588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c53960-5972-4977-9059-05b99d5897a3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.625099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a54a798-2a5f-4be2-93ad-b1fd1c4f738b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.628085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.628692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.628967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.629237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.629730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.629851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.630644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.630905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.631171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.631802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.631907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.634329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.642766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 810/1579] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:51.643622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.653044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafed11-7acd-4ef5-adba-8a2362c0b8fd admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.657513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.658144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.658439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.665911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] PATH_INFO: `/v3/projects/3f3deef8b1cc4c9193e9357f0ba99e15/users/835a297b6e75416ea9bd1a82b466b2a9/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.673508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f3deef8b1cc4c9193e9357f0ba99e15, user_id=835a297b6e75416ea9bd1a82b466b2a9, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.674587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:51:51.674587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:51:51.674587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:51:51.676558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.740991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 770/1580] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/users => generated 290 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.741844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.752251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facfce8a-fa21-4859-be20-1423bdcbd2c3 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.752819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 811/1581] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:51 2022] PUT /identity/v3/projects/3f3deef8b1cc4c9193e9357f0ba99e15/users/835a297b6e75416ea9bd1a82b466b2a9/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:51.753685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.811701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.812251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:51.812870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.812870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.813159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.813735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.815684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.817280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51db0632-d479-4b62-9767-788facb5d515 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.821886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.823601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.823601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.823601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.823954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.825864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.912067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 771/1582] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/users => generated 285 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.913312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.923731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23619ef-d28a-4442-8cfd-bdda75c1105c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.928439 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.929021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.929333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.929640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.930392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.932002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.934212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:51.941315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 812/1583] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/users => generated 287 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:51.942219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.947332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 772/1584] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:51.947332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.952673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53103f8c-eb24-416b-bd59-91724670aca4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.954003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392d4ce9-a3bf-4d39-a4c4-2d64b818b9c7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.957495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.958088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.958368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.958583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.958707 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.959160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.959270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.959438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.959691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.960301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.961403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.963904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:51.973165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 813/1585] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:51.974243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:51.986985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3da545-e034-4b7e-abdc-1193da7daab8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:51.991549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:51.999935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:51.999935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:51.999935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:51.999935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:51.999935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.005819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 814/1586] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.006688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.016612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d92af0-8cfc-487b-b0cd-600e2d98a588 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.021248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2c9454-3582-4332-802d-10d70161f298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.021821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c9454-3582-4332-802d-10d70161f298 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.022096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c9454-3582-4332-802d-10d70161f298 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.022364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2c9454-3582-4332-802d-10d70161f298 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.023029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2c9454-3582-4332-802d-10d70161f298 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.025305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2c9454-3582-4332-802d-10d70161f298 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.035176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 815/1587] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.036356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30421665-22ac-45f0-9b32-4313891092bd None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.036637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30421665-22ac-45f0-9b32-4313891092bd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.036988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30421665-22ac-45f0-9b32-4313891092bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.038834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 773/1588] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:51 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:52.038834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.049980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c753ed2a-2da9-42d4-ae8c-8cde3956a093 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.054751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.055345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.055619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.055884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.056515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.057481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-30421665-22ac-45f0-9b32-4313891092bd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:52.058094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.058716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-30421665-22ac-45f0-9b32-4313891092bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:52.066763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 816/1589] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:52.067525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30421665-22ac-45f0-9b32-4313891092bd None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.070982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 774/1590] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.070982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.077108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30421665-22ac-45f0-9b32-4313891092bd None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.081816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.083594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.083594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.083594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.083850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.085424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.086421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c99d3c8-9855-46db-be61-cb362da103ff admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.090962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.091559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.091829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.092095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.092854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.094449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.096008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 817/1591] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.096781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.107001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989043b1-89eb-450c-8840-625d465875ed admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.109313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 775/1592] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.109968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.115345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.116211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.116211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.116483 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] PATH_INFO: `/v3/projects/fcf59d2f167f446ca5ce0038a3ca3dff/users/c2e9337e80b44ab291d78bf5295b56b2/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.119905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc55105-fb63-45cb-8553-da132153e708 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.124531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.129463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.129463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.129463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] PATH_INFO: `/v3/projects/24b4590edfe642bc8a8bd15a1b8ba464/users/185e359e571f44bda1585124529b36a8/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.131871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcf59d2f167f446ca5ce0038a3ca3dff, user_id=c2e9337e80b44ab291d78bf5295b56b2, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.134721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.140837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=24b4590edfe642bc8a8bd15a1b8ba464, user_id=185e359e571f44bda1585124529b36a8, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.143648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.210640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 818/1593] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:52 2022] PUT /identity/v3/projects/fcf59d2f167f446ca5ce0038a3ca3dff/users/c2e9337e80b44ab291d78bf5295b56b2/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:52.212185 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.238486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 776/1594] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:52 2022] PUT /identity/v3/projects/24b4590edfe642bc8a8bd15a1b8ba464/users/185e359e571f44bda1585124529b36a8/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:52.244403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.279354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99aed791-07ed-41db-a378-238410234df8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.305765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fe2b43-b8ab-444c-91c4-d580f7eafcbf admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.310585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.311252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.311529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.311803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.312438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.314016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.325427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 777/1595] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.326788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.335579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.336918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.337199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.337466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.338112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.338924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d0fa8-9b3f-42ac-a119-3b988f95a1a2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.339751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.343539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.344128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.344398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.344667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.345367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.346963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.349894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 819/1596] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.350753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.357898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 778/1597] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.358755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.367320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:52.368262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007460de-97bb-40b4-a7ec-7bcc7cc2c000 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.372900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec73ca1f-86b1-4afd-ab08-85b60dd57431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.373629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec73ca1f-86b1-4afd-ab08-85b60dd57431 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.379656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec73ca1f-86b1-4afd-ab08-85b60dd57431 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.379656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec73ca1f-86b1-4afd-ab08-85b60dd57431 admin admin] PATH_INFO: `/v3/projects/8760be982f804a9aa319b989aa1c3118/users/14d56042415540a3b3ed06578b712ca5/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.390427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec73ca1f-86b1-4afd-ab08-85b60dd57431 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8760be982f804a9aa319b989aa1c3118, user_id=14d56042415540a3b3ed06578b712ca5, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.393187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec73ca1f-86b1-4afd-ab08-85b60dd57431 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.412792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5916733-8c97-4ca4-a4a1-ffa8acd984f8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.416607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.417212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.417491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.417762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.426982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.428424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.438084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 820/1598] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:52.438921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.448235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71685ea5-fac2-400e-9770-b428be9b8010 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.452847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3fad1-f0be-493e-9c83-84fff61a1707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.453436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3fad1-f0be-493e-9c83-84fff61a1707 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.453714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3fad1-f0be-493e-9c83-84fff61a1707 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.453991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3fad1-f0be-493e-9c83-84fff61a1707 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.454663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3fad1-f0be-493e-9c83-84fff61a1707 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.456515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3fad1-f0be-493e-9c83-84fff61a1707 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.462445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 779/1599] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:52 2022] PUT /identity/v3/projects/8760be982f804a9aa319b989aa1c3118/users/14d56042415540a3b3ed06578b712ca5/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:52.463830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf6fa1-df53-4afe-8071-3d4080e3499f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.464555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf6fa1-df53-4afe-8071-3d4080e3499f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.464959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cf6fa1-df53-4afe-8071-3d4080e3499f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.467811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 821/1600] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.471459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51af66-e8b7-4b56-bc31-303ea6992faa None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.471459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51af66-e8b7-4b56-bc31-303ea6992faa None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.471459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51af66-e8b7-4b56-bc31-303ea6992faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.491034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ab51af66-e8b7-4b56-bc31-303ea6992faa None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:52.492320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ab51af66-e8b7-4b56-bc31-303ea6992faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:52.493760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-28cf6fa1-df53-4afe-8071-3d4080e3499f None None] MFA Rules not processed for user `c2e9337e80b44ab291d78bf5295b56b2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:52.495009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-28cf6fa1-df53-4afe-8071-3d4080e3499f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:52.540031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:52.553507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 822/1601] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:52.558304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51af66-e8b7-4b56-bc31-303ea6992faa None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.564796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51af66-e8b7-4b56-bc31-303ea6992faa None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.565156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 780/1602] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:52 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:52.566418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf6fa1-df53-4afe-8071-3d4080e3499f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.572089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.572879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.573168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.573442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.574095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.575742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.581068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cf6fa1-df53-4afe-8071-3d4080e3499f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.587252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 823/1603] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:52.587950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04333c16-9f33-4827-ab99-669e3f9356ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.591945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.592056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04333c16-9f33-4827-ab99-669e3f9356ed admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.592056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04333c16-9f33-4827-ab99-669e3f9356ed admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.592056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04333c16-9f33-4827-ab99-669e3f9356ed admin admin] PATH_INFO: `/v3/projects/084b278155af4b28819357bcf190522d/users/1dcf55e151b94d379bcf21073f574af5/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.604857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc7ca3d-135e-4fbf-a5b1-02b40cb3d8e9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.608001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04333c16-9f33-4827-ab99-669e3f9356ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=084b278155af4b28819357bcf190522d, user_id=1dcf55e151b94d379bcf21073f574af5, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.609861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.610663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.610804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04333c16-9f33-4827-ab99-669e3f9356ed admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.610906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.611149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] PATH_INFO: `/v3/projects/14861bfe355b443c9a3be140e2990a80/users/8b0946e88f4a4133a6b2cf6e3602732b/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.627644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14861bfe355b443c9a3be140e2990a80, user_id=8b0946e88f4a4133a6b2cf6e3602732b, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.630825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.673165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 781/1604] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:52 2022] PUT /identity/v3/projects/084b278155af4b28819357bcf190522d/users/1dcf55e151b94d379bcf21073f574af5/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:52.674493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769906de-d383-48c1-a289-f0466b166f34 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.674778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769906de-d383-48c1-a289-f0466b166f34 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.675051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769906de-d383-48c1-a289-f0466b166f34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.693822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-769906de-d383-48c1-a289-f0466b166f34 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:52.696713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-769906de-d383-48c1-a289-f0466b166f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:52.718431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 824/1605] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:52 2022] PUT /identity/v3/projects/14861bfe355b443c9a3be140e2990a80/users/8b0946e88f4a4133a6b2cf6e3602732b/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:52.719752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.771819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 782/1606] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:52.771819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769906de-d383-48c1-a289-f0466b166f34 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.794115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769906de-d383-48c1-a289-f0466b166f34 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.798576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.799488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.799583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea7d64-19bf-4d08-8f3a-47068254a177 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:52.799678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.799968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.800573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.802725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.805794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:52.806686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:52.806686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:52.807034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] PATH_INFO: `/v3/projects/d2581751051d45deb15e3461411ec41e/users/06065e7f3625480db47ef2f3e53aff85/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:52.825226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d2581751051d45deb15e3461411ec41e, user_id=06065e7f3625480db47ef2f3e53aff85, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:52.827911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:52.830180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:52.905980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 783/1607] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:52 2022] POST /identity/v3/projects => generated 403 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:52.905980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.907300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:52.929273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 825/1608] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:52 2022] PUT /identity/v3/projects/d2581751051d45deb15e3461411ec41e/users/06065e7f3625480db47ef2f3e53aff85/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:52.931211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:52.970522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68d92bc-d192-4c4a-bdce-c69b5aae54c0 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.017305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77d9973-3d6d-476c-9744-9853a3b394b2 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.024331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.025493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.026001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.026538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] PATH_INFO: `/v3/projects/5242e7f780ff4ca1adc24ea6c21041e1/users/e6f0b08161bf408888e0f9886746c9f9/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.028317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ec7f95-105e-4c81-a478-3ab5046195f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.029345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec7f95-105e-4c81-a478-3ab5046195f4 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.029809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec7f95-105e-4c81-a478-3ab5046195f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.030087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec7f95-105e-4c81-a478-3ab5046195f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.045078 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec7f95-105e-4c81-a478-3ab5046195f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.046810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec7f95-105e-4c81-a478-3ab5046195f4 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.048918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5242e7f780ff4ca1adc24ea6c21041e1, user_id=e6f0b08161bf408888e0f9886746c9f9, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.051310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:53.052324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.058715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 784/1609] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:53.060085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5758377-f584-4a37-a31c-e6b2cb01305d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.060534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5758377-f584-4a37-a31c-e6b2cb01305d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.060831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5758377-f584-4a37-a31c-e6b2cb01305d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.089984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d5758377-f584-4a37-a31c-e6b2cb01305d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:53.091544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d5758377-f584-4a37-a31c-e6b2cb01305d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:53.099585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 785/1610] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:53.100773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b33bd6-8a38-47e9-b272-b50659c58d22 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.101056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b33bd6-8a38-47e9-b272-b50659c58d22 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.101324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b33bd6-8a38-47e9-b272-b50659c58d22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.129778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-87b33bd6-8a38-47e9-b272-b50659c58d22 None None] MFA Rules not processed for user `8b0946e88f4a4133a6b2cf6e3602732b`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:53.131281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-87b33bd6-8a38-47e9-b272-b50659c58d22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:53.152427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 826/1611] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:52 2022] PUT /identity/v3/projects/5242e7f780ff4ca1adc24ea6c21041e1/users/e6f0b08161bf408888e0f9886746c9f9/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:53.157724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.218477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 786/1612] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:53.222396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b33bd6-8a38-47e9-b272-b50659c58d22 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.233411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b33bd6-8a38-47e9-b272-b50659c58d22 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.248123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53896d2b-0467-4f21-91df-437613d0365d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.252863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87438c6-8600-4558-9da6-f53b189de9f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.253613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87438c6-8600-4558-9da6-f53b189de9f3 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.253900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87438c6-8600-4558-9da6-f53b189de9f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.254322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87438c6-8600-4558-9da6-f53b189de9f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.306062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.306623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.306908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.307178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.307816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.309762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.338140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87438c6-8600-4558-9da6-f53b189de9f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.339711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87438c6-8600-4558-9da6-f53b189de9f3 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.352099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 827/1613] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:53.353298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec37975-04b8-4d8d-af19-a1665df41537 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.353783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec37975-04b8-4d8d-af19-a1665df41537 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.355819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec37975-04b8-4d8d-af19-a1665df41537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.397236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-cec37975-04b8-4d8d-af19-a1665df41537 None None] MFA Rules not processed for user `06065e7f3625480db47ef2f3e53aff85`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:53.402951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cec37975-04b8-4d8d-af19-a1665df41537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:53.413074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:53.486212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 787/1614] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/users => generated 308 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:53.489595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.502601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 828/1615] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:53.504065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718c6864-96bd-4fe0-b380-99471b0f83bf None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.506763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718c6864-96bd-4fe0-b380-99471b0f83bf None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.506763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718c6864-96bd-4fe0-b380-99471b0f83bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.513981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed34f1fc-7d66-43f7-a766-6c4d7833162b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.522347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.527698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.527698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.527698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.551146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.562195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.579122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-718c6864-96bd-4fe0-b380-99471b0f83bf None None] MFA Rules not processed for user `e6f0b08161bf408888e0f9886746c9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:53.580368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-718c6864-96bd-4fe0-b380-99471b0f83bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:53.581863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 788/1616] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:53.582767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.602573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc1185e-f341-485e-800f-80ab8d3f90d4 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.608793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.609725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.610143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.610603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.611278 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.617999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.662332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 829/1617] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:53.666877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718c6864-96bd-4fe0-b380-99471b0f83bf None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.677337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718c6864-96bd-4fe0-b380-99471b0f83bf None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.682498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508dc61d-7ce4-4dcf-9705-921bc25da465 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.683108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508dc61d-7ce4-4dcf-9705-921bc25da465 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.683446 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508dc61d-7ce4-4dcf-9705-921bc25da465 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.683674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508dc61d-7ce4-4dcf-9705-921bc25da465 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.684591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508dc61d-7ce4-4dcf-9705-921bc25da465 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.686189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508dc61d-7ce4-4dcf-9705-921bc25da465 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.697482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 830/1618] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:53 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:53.698737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e827f823-ab78-4cd8-ab4b-04386318c748 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.699019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e827f823-ab78-4cd8-ab4b-04386318c748 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.705681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e827f823-ab78-4cd8-ab4b-04386318c748 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.705681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 831/1619] 10.222.0.27 () {54 vars in 961 bytes} [Tue Oct 4 08:51:53 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:51:53.705681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e827f823-ab78-4cd8-ab4b-04386318c748 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.714727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e827f823-ab78-4cd8-ab4b-04386318c748 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.719827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f314ccf-2414-4d58-a2d9-2239b37cb681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.720523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f314ccf-2414-4d58-a2d9-2239b37cb681 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.720876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f314ccf-2414-4d58-a2d9-2239b37cb681 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.721219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f314ccf-2414-4d58-a2d9-2239b37cb681 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.723540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f314ccf-2414-4d58-a2d9-2239b37cb681 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:53.725590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f314ccf-2414-4d58-a2d9-2239b37cb681 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:53.763055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 789/1620] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/projects => generated 367 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:53.763055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.816042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:53.840076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 832/1621] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/projects => generated 371 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:53.845628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d10b7-14e6-4916-9298-fca62f8744e4 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.846075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d10b7-14e6-4916-9298-fca62f8744e4 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.846427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281d10b7-14e6-4916-9298-fca62f8744e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:53.849980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370e0b4d-a44d-48b9-89c3-93e160732f78 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:53.933858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-281d10b7-14e6-4916-9298-fca62f8744e4 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:53.935877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-281d10b7-14e6-4916-9298-fca62f8744e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:53.968064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:53.968745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:53.969024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:53.969305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] PATH_INFO: `/v3/projects/87f8d8c6504147fda945fca06f6ec945/users/7fcee15930574c89909dceba71a4d517/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.013872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87f8d8c6504147fda945fca06f6ec945, user_id=7fcee15930574c89909dceba71a4d517, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.016868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.037555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:54.058340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 833/1622] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Oct 4 08:51:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:54.059716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281d10b7-14e6-4916-9298-fca62f8744e4 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.073506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281d10b7-14e6-4916-9298-fca62f8744e4 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.078521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.079531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.079882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.080257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.081025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.084144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.141523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 790/1623] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:53 2022] PUT /identity/v3/projects/87f8d8c6504147fda945fca06f6ec945/users/7fcee15930574c89909dceba71a4d517/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:54.146386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.197292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 834/1624] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:54 2022] POST /identity/v3/users => generated 290 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:54.197292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.228862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d15f36e-6e0e-46da-8cab-bc3cb7f6ee96 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.230672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a2824f-3d6c-4439-8da6-7eaf80cb5451 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.235140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.236005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.236357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.236631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.237507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.240720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.321167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.321167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.321545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.321769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.348573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 791/1625] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:51:54 2022] POST /identity/v3/users => generated 292 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:51:54.348573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.364963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbf87d9-6a1c-4437-89ec-06a1c0b29767 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.378782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.379720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.379957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.380543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.381272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.383904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.407527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 792/1626] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:54 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:54.408510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.421980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c9a78b-45fa-416f-97f0-c5f090b97397 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.427253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.431104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.431104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.431104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.431104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.431507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.447900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 793/1627] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:54 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:54.448834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.460597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.463205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.473345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bdcf2-5aeb-4e6b-99d4-999cb3f5d3a5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.474813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 835/1628] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:54.476041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.484402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.486151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.486441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.486719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.487425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.489033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958ebb6-429d-4c3b-aab0-9303f1260ff4 service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.492787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.495981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.496931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.497787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.498249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] PATH_INFO: `/v3/projects/87f8d8c6504147fda945fca06f6ec945/users/7fcee15930574c89909dceba71a4d517/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.501473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87f8d8c6504147fda945fca06f6ec945, user_id=7fcee15930574c89909dceba71a4d517, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.506149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.509140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 794/1629] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:54 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:54.510088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.520845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 836/1630] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:54 2022] PUT /identity/v3/projects/87f8d8c6504147fda945fca06f6ec945/users/7fcee15930574c89909dceba71a4d517/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:54.521156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5908c8d-be7d-4854-accb-278c69c4ca3d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.522785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.616579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4978955a-b17c-4d95-9019-11e943c1aa5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.617398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4978955a-b17c-4d95-9019-11e943c1aa5b admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.617876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4978955a-b17c-4d95-9019-11e943c1aa5b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.618291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4978955a-b17c-4d95-9019-11e943c1aa5b admin admin] PATH_INFO: `/v3/projects/ccdd7ea571124358b17262fec27f5cf2/users/a7c7b5c7f46048f5bc1cc4fa245d636a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.630506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2b3caa-bd31-4a47-86a7-2f81c3b82a57 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.639128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.639810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.639937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:54.640145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.640362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] PATH_INFO: `/v3/projects/f6d845930c7a4c39af048eff19871f04/users/8533b7a4cc194bb697c6798186988d6d/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.644049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4978955a-b17c-4d95-9019-11e943c1aa5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccdd7ea571124358b17262fec27f5cf2, user_id=a7c7b5c7f46048f5bc1cc4fa245d636a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.647133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4978955a-b17c-4d95-9019-11e943c1aa5b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.660924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6d845930c7a4c39af048eff19871f04, user_id=8533b7a4cc194bb697c6798186988d6d, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.670787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.671310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:54.722751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 795/1631] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:54 2022] PUT /identity/v3/projects/ccdd7ea571124358b17262fec27f5cf2/users/a7c7b5c7f46048f5bc1cc4fa245d636a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:54.722751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f4e56a-c1a1-423e-abf8-fdf00b704385 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.722979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f4e56a-c1a1-423e-abf8-fdf00b704385 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.723266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f4e56a-c1a1-423e-abf8-fdf00b704385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.749397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 837/1632] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:54 2022] PUT /identity/v3/projects/f6d845930c7a4c39af048eff19871f04/users/8533b7a4cc194bb697c6798186988d6d/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:54.750212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.794599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-08f4e56a-c1a1-423e-abf8-fdf00b704385 None None] MFA Rules not processed for user `7fcee15930574c89909dceba71a4d517`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:54.796672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-08f4e56a-c1a1-423e-abf8-fdf00b704385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:54.881975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201fb6fb-e892-4787-8370-08a79256182a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.893893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.895481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.896489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.896489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.897883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.900640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.933987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 838/1633] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:54 2022] GET /identity/v3/roles => generated 1420 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:54.935341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.950250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0527cb-6263-4d7b-b6c4-f251245401f3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.964509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 796/1634] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:54 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:54.964509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f4e56a-c1a1-423e-abf8-fdf00b704385 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:54.964685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.964685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.964685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.968573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.969379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.971713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.973724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f4e56a-c1a1-423e-abf8-fdf00b704385 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.979684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.980561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.980849 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.981123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] PATH_INFO: `/v3/projects/ccdd7ea571124358b17262fec27f5cf2/users/a7c7b5c7f46048f5bc1cc4fa245d636a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.983401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccdd7ea571124358b17262fec27f5cf2, user_id=a7c7b5c7f46048f5bc1cc4fa245d636a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.986984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:55.006688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 839/1635] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:51:54 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:51:55.009874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:55.019194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 797/1636] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:54 2022] PUT /identity/v3/projects/ccdd7ea571124358b17262fec27f5cf2/users/a7c7b5c7f46048f5bc1cc4fa245d636a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:55.033973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:55.146841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b0274a-7666-4b4b-92eb-94e4c0aed224 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:55.205096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52476e8-bca0-468b-94ed-e8758d03d32a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:55.224113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c4216e-56a0-4c4c-a641-321697c736a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:55.225501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c4216e-56a0-4c4c-a641-321697c736a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.225976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c4216e-56a0-4c4c-a641-321697c736a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.226260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c4216e-56a0-4c4c-a641-321697c736a5 admin admin] PATH_INFO: `/v3/projects/f6d845930c7a4c39af048eff19871f04/users/8533b7a4cc194bb697c6798186988d6d/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.230702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3d0723-6a88-4660-8bac-c625a42c7357 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:55.231377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3d0723-6a88-4660-8bac-c625a42c7357 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.231662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3d0723-6a88-4660-8bac-c625a42c7357 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.231933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3d0723-6a88-4660-8bac-c625a42c7357 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.234718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c4216e-56a0-4c4c-a641-321697c736a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6d845930c7a4c39af048eff19871f04, user_id=8533b7a4cc194bb697c6798186988d6d, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:55.237782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c4216e-56a0-4c4c-a641-321697c736a5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:55.267868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 798/1637] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:51:55 2022] PUT /identity/v3/projects/f6d845930c7a4c39af048eff19871f04/users/8533b7a4cc194bb697c6798186988d6d/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:55.511492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4be6a5-fb41-4185-95a0-e36b78755ad8 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.511492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4be6a5-fb41-4185-95a0-e36b78755ad8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.511492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4be6a5-fb41-4185-95a0-e36b78755ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.511492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:55.514966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3d0723-6a88-4660-8bac-c625a42c7357 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:55.514966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3d0723-6a88-4660-8bac-c625a42c7357 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:55.516780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8b4be6a5-fb41-4185-95a0-e36b78755ad8 None None] MFA Rules not processed for user `a7c7b5c7f46048f5bc1cc4fa245d636a`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:55.516780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8b4be6a5-fb41-4185-95a0-e36b78755ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:55.517109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 840/1638] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:55 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:55.517109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358beb34-c7b7-449f-a325-b3494ba462f3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.517109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358beb34-c7b7-449f-a325-b3494ba462f3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.517109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358beb34-c7b7-449f-a325-b3494ba462f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.517109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:55.517109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-358beb34-c7b7-449f-a325-b3494ba462f3 None None] MFA Rules not processed for user `8533b7a4cc194bb697c6798186988d6d`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:55.519973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-358beb34-c7b7-449f-a325-b3494ba462f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:55.613259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 799/1639] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:55 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:55.637787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4be6a5-fb41-4185-95a0-e36b78755ad8 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:55.724599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 841/1640] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:55 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:55.750918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67474f78-6efe-490f-a1e4-69ae6d945fdd None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.751198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67474f78-6efe-490f-a1e4-69ae6d945fdd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.751479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67474f78-6efe-490f-a1e4-69ae6d945fdd None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.753131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 842/1641] 10.222.0.27 () {54 vars in 961 bytes} [Tue Oct 4 08:51:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:51:55.766141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a686e-410b-4ec9-98cb-f77ccb5903f0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.766431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a686e-410b-4ec9-98cb-f77ccb5903f0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.766757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a686e-410b-4ec9-98cb-f77ccb5903f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.806032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4be6a5-fb41-4185-95a0-e36b78755ad8 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:55.811636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-4f5a686e-410b-4ec9-98cb-f77ccb5903f0 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:55.814184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4f5a686e-410b-4ec9-98cb-f77ccb5903f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:55.816397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:55.817557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.817823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.818712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.838123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:55.840840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:55.877800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 800/1642] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:55 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:55.878838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:55.907110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1dd8b-e875-4ae7-ae48-409c41aa1926 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:55.926843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3769edbe-166b-430b-a7c4-3fb853b95ca8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:55.927475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769edbe-166b-430b-a7c4-3fb853b95ca8 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:55.927751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769edbe-166b-430b-a7c4-3fb853b95ca8 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:55.928023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3769edbe-166b-430b-a7c4-3fb853b95ca8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:55.946090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3769edbe-166b-430b-a7c4-3fb853b95ca8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:55.947688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3769edbe-166b-430b-a7c4-3fb853b95ca8 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:55.977119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 801/1643] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:55 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:56.007109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 843/1644] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Oct 4 08:51:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:56.009155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89471f53-f323-47b5-8e28-b172fcfb01d7 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.009526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89471f53-f323-47b5-8e28-b172fcfb01d7 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.014921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89471f53-f323-47b5-8e28-b172fcfb01d7 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.014921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 802/1645] 10.222.0.27 () {54 vars in 912 bytes} [Tue Oct 4 08:51:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:51:56.027728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc1d558-3272-47c9-b6d3-58b98954d082 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.028717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc1d558-3272-47c9-b6d3-58b98954d082 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.029201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc1d558-3272-47c9-b6d3-58b98954d082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.075814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-2fc1d558-3272-47c9-b6d3-58b98954d082 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:56.077146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2fc1d558-3272-47c9-b6d3-58b98954d082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:56.099213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89471f53-f323-47b5-8e28-b172fcfb01d7 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:56.104983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 844/1646] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:51:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:56.138382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc1d558-3272-47c9-b6d3-58b98954d082 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:56.152260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc1d558-3272-47c9-b6d3-58b98954d082 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:56.159106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799c2bf6-9db2-4efa-819f-7b2c9426cd32 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:56.159875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799c2bf6-9db2-4efa-819f-7b2c9426cd32 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.160223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799c2bf6-9db2-4efa-819f-7b2c9426cd32 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.160809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799c2bf6-9db2-4efa-819f-7b2c9426cd32 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.175866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799c2bf6-9db2-4efa-819f-7b2c9426cd32 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:56.179152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799c2bf6-9db2-4efa-819f-7b2c9426cd32 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:56.195832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 845/1647] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:51:56.231947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89471f53-f323-47b5-8e28-b172fcfb01d7 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:56.251229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:56.251911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.252191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.252466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] PATH_INFO: `/v3/users/c2e9337e80b44ab291d78bf5295b56b2` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.253859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2e9337e80b44ab291d78bf5295b56b2)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:56.256088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:56.421749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435739d3-b974-4fe4-a649-be1123b39ec5 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.422494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435739d3-b974-4fe4-a649-be1123b39ec5 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.422965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435739d3-b974-4fe4-a649-be1123b39ec5 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.425109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 846/1648] 10.222.0.27 () {54 vars in 912 bytes} [Tue Oct 4 08:51:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:51:56.427882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:56.432792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc0551-16ba-4285-9a84-102ed67c7935 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.433103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc0551-16ba-4285-9a84-102ed67c7935 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.433347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc0551-16ba-4285-9a84-102ed67c7935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.466820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-cddc0551-16ba-4285-9a84-102ed67c7935 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:56.469137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cddc0551-16ba-4285-9a84-102ed67c7935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:56.480574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 847/1649] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:51:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:51:56.523127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc0551-16ba-4285-9a84-102ed67c7935 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:56.537743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc0551-16ba-4285-9a84-102ed67c7935 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:56.540885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2e9337e80b44ab291d78bf5295b56b2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:56.553487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5c994e-577c-4cca-b0d5-7361f395b2ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:56.553487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5c994e-577c-4cca-b0d5-7361f395b2ad service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.553487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5c994e-577c-4cca-b0d5-7361f395b2ad service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.553487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5c994e-577c-4cca-b0d5-7361f395b2ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.566278 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2e9337e80b44ab291d78bf5295b56b2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:56.566962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2e9337e80b44ab291d78bf5295b56b2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:56.567857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2e9337e80b44ab291d78bf5295b56b2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:56.583622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5c994e-577c-4cca-b0d5-7361f395b2ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:56.589859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5c994e-577c-4cca-b0d5-7361f395b2ad service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:56.614751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 848/1650] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:56.686872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2e9337e80b44ab291d78bf5295b56b2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:56.738394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 803/1651] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:51:56 2022] DELETE /identity/v3/users/c2e9337e80b44ab291d78bf5295b56b2 => generated 0 bytes in 640 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:51:56.743578 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.743860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.744129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.801545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:56.805083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:56.812513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:56.905070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:56.906651 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 849/1652] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:56.906916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:56.942791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:56.944644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:56.949841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c08209a-c54f-4d96-8733-25081cf21b22 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:56.963233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:56.965887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:56.965887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:56.965887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] PATH_INFO: `/v3/users/06065e7f3625480db47ef2f3e53aff85` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:56.967305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=06065e7f3625480db47ef2f3e53aff85)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:56.981271 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:57.058212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec2ed2-1fa5-4c6d-a00b-c56964c4c658 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:57.062581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2434058-6753-489f-9113-46c886a1b55b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:57.063430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2434058-6753-489f-9113-46c886a1b55b service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:57.063908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2434058-6753-489f-9113-46c886a1b55b service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:57.064245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2434058-6753-489f-9113-46c886a1b55b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:57.080606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2434058-6753-489f-9113-46c886a1b55b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:57.081659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2434058-6753-489f-9113-46c886a1b55b service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:57.094025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 850/1653] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:51:57.238822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06065e7f3625480db47ef2f3e53aff85'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:57.254055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06065e7f3625480db47ef2f3e53aff85'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:57.269896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06065e7f3625480db47ef2f3e53aff85'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:57.271369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06065e7f3625480db47ef2f3e53aff85'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:57.394779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06065e7f3625480db47ef2f3e53aff85'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:57.426800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 804/1654] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:51:56 2022] DELETE /identity/v3/users/06065e7f3625480db47ef2f3e53aff85 => generated 0 bytes in 525 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:57.428843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:57.430999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:57.447468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:57.513974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:57.523743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:57.650533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 851/1655] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:51:57.673507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:57.681106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:57.826122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367de67e-2c49-4da0-8668-85a4050ea5bd admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:57.832335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:57.833147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:57.833532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:57.834588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:57.846462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:57.848100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:57.873406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 805/1656] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:57.921745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:57.925097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:57.949877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50a4ab6-e154-468d-a526-222e9da19a93 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:57.960024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:57.961305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:57.962516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:57.968450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] PATH_INFO: `/v3/users/e6f0b08161bf408888e0f9886746c9f9` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:57.968450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6f0b08161bf408888e0f9886746c9f9)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:57.971006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: WARNING py.warnings [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:51:57.971006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: warnings.warn(msg) Oct 04 08:51:57.971006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: Oct 04 08:51:57.984027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:58.380998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:58.385717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:58.506858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06495e73-9ef1-4f29-8b2a-197733a987e0 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:58.509559 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6f0b08161bf408888e0f9886746c9f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.516448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:58.524296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:58.526586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:58.527087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:58.527567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] PATH_INFO: `/v3/projects/fcf59d2f167f446ca5ce0038a3ca3dff` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:58.532748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fcf59d2f167f446ca5ce0038a3ca3dff)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:58.534129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: WARNING py.warnings [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:51:58.534129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: warnings.warn(msg) Oct 04 08:51:58.534129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: Oct 04 08:51:58.536017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:58.567269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6f0b08161bf408888e0f9886746c9f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.570855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6f0b08161bf408888e0f9886746c9f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.574618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6f0b08161bf408888e0f9886746c9f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.620238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:58.651541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b12c857d-8d0f-45db-b97c-6bd029c3ab48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6f0b08161bf408888e0f9886746c9f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.713934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 852/1657] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:51:57 2022] DELETE /identity/v3/users/e6f0b08161bf408888e0f9886746c9f9 => generated 0 bytes in 793 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:58.718674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59c237e-bea6-4692-8327-827715443f8e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:58.719135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59c237e-bea6-4692-8327-827715443f8e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:58.719429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59c237e-bea6-4692-8327-827715443f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:58.764710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f59c237e-bea6-4692-8327-827715443f8e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:58.771880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f59c237e-bea6-4692-8327-827715443f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:58.812640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fcf59d2f167f446ca5ce0038a3ca3dff', 'request_id': 'req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.918095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fcf59d2f167f446ca5ce0038a3ca3dff', 'request_id': 'req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.924386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 853/1658] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:51:58.945712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fcf59d2f167f446ca5ce0038a3ca3dff', 'request_id': 'req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.973150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] The token cache is being invalidate because project fcf59d2f167f446ca5ce0038a3ca3dff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:51:58.973666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.974763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:58.979681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 806/1659] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:51:58 2022] DELETE /identity/v3/projects/fcf59d2f167f446ca5ce0038a3ca3dff => generated 0 bytes in 599 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:58.983202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59c237e-bea6-4692-8327-827715443f8e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:58.985354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f59c237e-bea6-4692-8327-827715443f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:59.018205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59c237e-bea6-4692-8327-827715443f8e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:59.040475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:59.041152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:59.041627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:59.041910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] PATH_INFO: `/v3/users/7fcee15930574c89909dceba71a4d517` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:59.045727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7fcee15930574c89909dceba71a4d517)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:59.052455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:59.065412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:59.067186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:59.281800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d42f9-1f0c-47f8-8503-ba2940c2b607 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:59.297042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:59.298046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:59.298614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:59.299632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:59.302180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:59.325810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:59.327919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:59.349324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 807/1660] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:59.399165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7fcee15930574c89909dceba71a4d517'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.415026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7fcee15930574c89909dceba71a4d517'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.416482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7fcee15930574c89909dceba71a4d517'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.419398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7fcee15930574c89909dceba71a4d517'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.474811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:59.485492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:59.512779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9907da4-8d31-4ecb-9c5f-f2cc4ffe8a89 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7fcee15930574c89909dceba71a4d517'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.519101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da20ead7-4fcf-45e6-9f64-131c43069460 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:59.552303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:59.555770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:59.556315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:59.556788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] PATH_INFO: `/v3/projects/d2581751051d45deb15e3461411ec41e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:59.557287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 854/1661] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:51:58 2022] DELETE /identity/v3/users/7fcee15930574c89909dceba71a4d517 => generated 0 bytes in 574 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:59.558843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d2581751051d45deb15e3461411ec41e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:59.566077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:59.780166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd2581751051d45deb15e3461411ec41e', 'request_id': 'req-56daa50c-0101-4f21-aa5d-802ed9b8b923'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.874384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd2581751051d45deb15e3461411ec41e', 'request_id': 'req-56daa50c-0101-4f21-aa5d-802ed9b8b923'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.889204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd2581751051d45deb15e3461411ec41e', 'request_id': 'req-56daa50c-0101-4f21-aa5d-802ed9b8b923'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.922648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] The token cache is being invalidate because project d2581751051d45deb15e3461411ec41e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:51:59.923359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.924185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-56daa50c-0101-4f21-aa5d-802ed9b8b923 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:51:59.927311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 808/1662] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:51:59 2022] DELETE /identity/v3/projects/d2581751051d45deb15e3461411ec41e => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:51:59.933384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23655a-4e9b-46be-b42c-18548de9d80b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:59.937155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23655a-4e9b-46be-b42c-18548de9d80b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:59.937155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23655a-4e9b-46be-b42c-18548de9d80b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:59.983226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-db23655a-4e9b-46be-b42c-18548de9d80b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:51:59.986456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-db23655a-4e9b-46be-b42c-18548de9d80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:00.098969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:00.147094 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 855/1663] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:51:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:00.155115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077947a5-fae6-4d12-8d43-5c9de53bf36a None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:00.155530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077947a5-fae6-4d12-8d43-5c9de53bf36a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:00.155977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077947a5-fae6-4d12-8d43-5c9de53bf36a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:00.196205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-077947a5-fae6-4d12-8d43-5c9de53bf36a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:00.197440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-077947a5-fae6-4d12-8d43-5c9de53bf36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:00.217193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 809/1664] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:00.222779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23655a-4e9b-46be-b42c-18548de9d80b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:00.235235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23655a-4e9b-46be-b42c-18548de9d80b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:00.244807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:00.245883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:00.246311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:00.246775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:00.248354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:00.248882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:00.250943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:00.286322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 856/1665] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:00.288758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077947a5-fae6-4d12-8d43-5c9de53bf36a None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:00.302097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077947a5-fae6-4d12-8d43-5c9de53bf36a None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:00.308002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:00.308853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:00.309282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:00.309685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:00.310561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:00.312645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:00.402494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:00.404449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:00.422909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:00.436934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 810/1666] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:00 2022] POST /identity/v3/projects => generated 387 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:00.437096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772932cc-bd70-4e10-97ac-bbeaf8d48683 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:00.437972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:00.562773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:00.563445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:00.563730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:00.564007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] PATH_INFO: `/v3/projects/5242e7f780ff4ca1adc24ea6c21041e1` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:00.565432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] RBAC: Authorizing `identity:delete_project(project_id=5242e7f780ff4ca1adc24ea6c21041e1)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:00.567176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d6fdb-9319-4979-a7d2-ab0f7926181d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:00.567555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:00.686759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:00.687409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:00.687693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:00.687965 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:00.688788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:00.690847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:00.791165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5242e7f780ff4ca1adc24ea6c21041e1', 'request_id': 'req-be8b8736-c929-4450-afec-692c3861669c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:00.810796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5242e7f780ff4ca1adc24ea6c21041e1', 'request_id': 'req-be8b8736-c929-4450-afec-692c3861669c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:00.869817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 811/1667] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:00 2022] POST /identity/v3/users => generated 294 bytes in 428 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:52:00.871061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:00.894225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269f1f40-ceb0-41e8-805c-f5100c2e1853 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:00.949342 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5242e7f780ff4ca1adc24ea6c21041e1', 'request_id': 'req-be8b8736-c929-4450-afec-692c3861669c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.003933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] The token cache is being invalidate because project 5242e7f780ff4ca1adc24ea6c21041e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:01.004227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.006251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.014103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 857/1668] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:00 2022] DELETE /identity/v3/projects/5242e7f780ff4ca1adc24ea6c21041e1 => generated 0 bytes in 609 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:01.015229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:01.020397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:01.116135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:01.120597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:01.123931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:01.125964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:01.126908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:01.148106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:01.177641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8b8736-c929-4450-afec-692c3861669c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:01.183761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:01.184638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:01.185113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:01.185539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] PATH_INFO: `/v3/users/8533b7a4cc194bb697c6798186988d6d` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:01.186892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8533b7a4cc194bb697c6798186988d6d)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:01.187020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 812/1669] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:01.189279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:01.190858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:01.193320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:01.210799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e165ed-3d55-4718-b6b3-49911bb63a33 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:01.222317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:01.223361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:01.223861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:01.224533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:01.225814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:01.229734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:01.246586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 813/1670] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:01 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:01.257778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:01.284297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cd855-dc58-45d5-b837-e82cdd34a49f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:01.298364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:01.298364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:01.298364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:01.298364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] PATH_INFO: `/v3/projects/cb42f476994e409aa5cd54dbc9331051/users/8b6d625d413c459bacdf9e4c8d278553/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:01.331925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb42f476994e409aa5cd54dbc9331051, user_id=8b6d625d413c459bacdf9e4c8d278553, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:01.358317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:01.448729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8533b7a4cc194bb697c6798186988d6d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.467396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8533b7a4cc194bb697c6798186988d6d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.468452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8533b7a4cc194bb697c6798186988d6d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.470643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8533b7a4cc194bb697c6798186988d6d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.496511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 814/1671] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:01 2022] PUT /identity/v3/projects/cb42f476994e409aa5cd54dbc9331051/users/8b6d625d413c459bacdf9e4c8d278553/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:01.512778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:01.515630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:01.529538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-efc23170-56de-4db9-8972-79fc6cfc10e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8533b7a4cc194bb697c6798186988d6d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:01.586749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 858/1672] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:01 2022] DELETE /identity/v3/users/8533b7a4cc194bb697c6798186988d6d => generated 0 bytes in 572 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:01.591314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:01.592545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:01.592545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:01.709516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] MFA Rules not processed for user `8b6d625d413c459bacdf9e4c8d278553`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:01.714458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:01.739196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fee2a4-4919-4547-bba1-8dc991c481b1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:01.797836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:01.897998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 859/1673] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:01 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:01.904852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:01.907197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:01.913498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:01.914389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:01.914880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:01.915168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] PATH_INFO: `/v3/users/8b0946e88f4a4133a6b2cf6e3602732b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:01.916389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b0946e88f4a4133a6b2cf6e3602732b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:01.918387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:01.983347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa79936-7e52-42a8-a39c-7283de80f555 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:02.007305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:02.008851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:02.011287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:02.011287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:02.011287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:02.012034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:02.064305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:02.139416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b0946e88f4a4133a6b2cf6e3602732b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.145433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 860/1674] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:01 2022] POST /identity/v3/projects => generated 385 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:02.154639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:02.167757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b0946e88f4a4133a6b2cf6e3602732b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.172181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b0946e88f4a4133a6b2cf6e3602732b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.173710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b0946e88f4a4133a6b2cf6e3602732b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.266935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b7a33fef-1b8f-4e1c-af22-1649ad4e7f17 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b0946e88f4a4133a6b2cf6e3602732b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.296418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b9c4f4-2b22-46b5-84bd-577bda3a7831 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:02.309141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:02.315202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:02.315202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:02.315202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:02.315202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:02.316363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 815/1675] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:01 2022] DELETE /identity/v3/users/8b0946e88f4a4133a6b2cf6e3602732b => generated 0 bytes in 804 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:02.316876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:02.322074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:02.322360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:02.322662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:02.390903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:02.392083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:02.496543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 861/1676] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:02 2022] POST /identity/v3/users => generated 299 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:02.501049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:02.503069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:02.529549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f8299-4f1c-433d-a6c7-90ea7f94ce9d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:02.575313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 816/1677] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:02.579201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:02.587472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:02.607459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b403c0-be63-4564-a304-ca53567a8c44 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:02.610846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:02.611796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:02.612266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:02.612640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] PATH_INFO: `/v3/projects/87f8d8c6504147fda945fca06f6ec945` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:02.614102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=87f8d8c6504147fda945fca06f6ec945)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:02.616452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:02.683911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:02.684609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:02.685334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:02.685334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:02.685715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:02.687353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:02.707914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 862/1678] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:02 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:02.708790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:02.713167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:02.742307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87f8d8c6504147fda945fca06f6ec945', 'request_id': 'req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.795576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7436ccb-9798-40e9-ad6c-f78da1b97d2a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:02.865256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87f8d8c6504147fda945fca06f6ec945', 'request_id': 'req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.877125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87f8d8c6504147fda945fca06f6ec945', 'request_id': 'req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.883898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f829a312-4e73-45ef-8035-2190d174fd04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:02.884603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f829a312-4e73-45ef-8035-2190d174fd04 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:02.884821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f829a312-4e73-45ef-8035-2190d174fd04 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:02.885084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f829a312-4e73-45ef-8035-2190d174fd04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:02.895621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] The token cache is being invalidate because project 87f8d8c6504147fda945fca06f6ec945 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:02.896837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.909083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:02.912322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 817/1679] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:02 2022] DELETE /identity/v3/projects/87f8d8c6504147fda945fca06f6ec945 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:02.913468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:02.917642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:03.045514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f829a312-4e73-45ef-8035-2190d174fd04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:03.048079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f829a312-4e73-45ef-8035-2190d174fd04 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:03.067287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 863/1680] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:03.070957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330146fe-c0fd-4290-8dad-473b8bf4e254 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.071711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330146fe-c0fd-4290-8dad-473b8bf4e254 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.072736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330146fe-c0fd-4290-8dad-473b8bf4e254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.120283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-330146fe-c0fd-4290-8dad-473b8bf4e254 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:03.122322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-330146fe-c0fd-4290-8dad-473b8bf4e254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:03.124960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773e25a5-53d0-4969-8e2f-6d2f7b5e8ad7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:03.140259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 864/1681] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:03.141611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:03.142320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.142627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.142900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] PATH_INFO: `/v3/projects/8d00871c71bd40e48425cb00d85c0d90/users/6b06a1bc014f41838b794b6f34db76a5/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.152069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dabf8d1-a9e2-4d27-8676-73f4f7bb9f02 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.152461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dabf8d1-a9e2-4d27-8676-73f4f7bb9f02 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.152989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dabf8d1-a9e2-4d27-8676-73f4f7bb9f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.179070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d00871c71bd40e48425cb00d85c0d90, user_id=6b06a1bc014f41838b794b6f34db76a5, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:03.182848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:03.196394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:03.199682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-5dabf8d1-a9e2-4d27-8676-73f4f7bb9f02 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:03.202568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5dabf8d1-a9e2-4d27-8676-73f4f7bb9f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:03.233716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 865/1682] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:03.235368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dabf8d1-a9e2-4d27-8676-73f4f7bb9f02 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:03.265669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dabf8d1-a9e2-4d27-8676-73f4f7bb9f02 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:03.274615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 818/1683] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:02 2022] PUT /identity/v3/projects/8d00871c71bd40e48425cb00d85c0d90/users/6b06a1bc014f41838b794b6f34db76a5/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:03.278609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:03.375485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:03.428339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:03.428339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.428339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.428339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.428855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:03.428855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:03.442641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:03.444987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30f77e6-8a87-4fc5-bb29-4562f53180a2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:03.483659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 866/1684] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:03.483834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:03.483834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.483834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.483834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.484149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:03.485570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:03.489803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:03.502802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c897f1-c8e6-4c17-9e2e-d6140b8265f3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:03.512842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:03.512842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.512842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.512842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.513746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:03.517305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:03.522623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 819/1685] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:03 2022] GET /identity/v3/roles => generated 1420 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:03.526300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:03.550354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfd99c8-9544-4bbb-8aa7-7746523014a8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:03.567660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f975410-810b-4e11-8952-5219d2ec0178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:03.568371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f975410-810b-4e11-8952-5219d2ec0178 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.568650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f975410-810b-4e11-8952-5219d2ec0178 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.568921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f975410-810b-4e11-8952-5219d2ec0178 admin admin] PATH_INFO: `/v3/projects/8d00871c71bd40e48425cb00d85c0d90/users/6b06a1bc014f41838b794b6f34db76a5/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.573116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f975410-810b-4e11-8952-5219d2ec0178 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d00871c71bd40e48425cb00d85c0d90, user_id=6b06a1bc014f41838b794b6f34db76a5, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:03.575861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f975410-810b-4e11-8952-5219d2ec0178 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:03.633455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 820/1686] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:03 2022] PUT /identity/v3/projects/8d00871c71bd40e48425cb00d85c0d90/users/6b06a1bc014f41838b794b6f34db76a5/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:03.643537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59ff56-d3b7-4017-9d56-eb5fd0bf071c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.643830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59ff56-d3b7-4017-9d56-eb5fd0bf071c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.644106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59ff56-d3b7-4017-9d56-eb5fd0bf071c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.647241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 867/1687] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:03 2022] POST /identity/v3/projects => generated 403 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:03.663332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:03.755839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5b59ff56-d3b7-4017-9d56-eb5fd0bf071c None None] MFA Rules not processed for user `6b06a1bc014f41838b794b6f34db76a5`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:03.759352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5b59ff56-d3b7-4017-9d56-eb5fd0bf071c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:03.879019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887a128b-15f7-488c-a3cd-cc321a88a147 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:03.885351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:03.885894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:03.892414 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:03.892853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:03.897452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:03.900079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.016681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 821/1688] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:03 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 374 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:04.025928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 868/1689] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:03 2022] POST /identity/v3/users => generated 302 bytes in 363 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:04.030637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.030970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b59ff56-d3b7-4017-9d56-eb5fd0bf071c None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.035657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:04.052542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b59ff56-d3b7-4017-9d56-eb5fd0bf071c None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.067110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.067746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.068027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.068301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.068959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:04.071760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.102799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 822/1690] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:04 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:04.106285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.133494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3822d318-fefd-44e2-9f72-9b222b02e3d4 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.149800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.150661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.155755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.155755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.155755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:04.158014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.189585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90599eab-97d9-41a1-a780-636003efe47c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.190083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 823/1691] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:04 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:04.194459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.196899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.197758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.198199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.198639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.209862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:04.212644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.221038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407bd302-d599-49a2-ac3a-d5ca3ee294a1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.224930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 869/1692] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:04.230499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dfb4d0-b0eb-40f6-a0cc-e3a924739e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.231686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dfb4d0-b0eb-40f6-a0cc-e3a924739e5e admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.232059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dfb4d0-b0eb-40f6-a0cc-e3a924739e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.232545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dfb4d0-b0eb-40f6-a0cc-e3a924739e5e admin admin] PATH_INFO: `/v3/projects/15b0d1bf85e0404282622a3a3213d2c4/users/7cdead2514334b62afa46f4e61f14e96/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.251986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dfb4d0-b0eb-40f6-a0cc-e3a924739e5e admin admin] RBAC: Authorizing `identity:create_grant(project_id=15b0d1bf85e0404282622a3a3213d2c4, user_id=7cdead2514334b62afa46f4e61f14e96, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:04.255567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dfb4d0-b0eb-40f6-a0cc-e3a924739e5e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.285044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.299923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250f3e59-9d26-4770-ab5f-929b821d2af6 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.305932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.306770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.307741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.308125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] PATH_INFO: `/v3/users/8b6d625d413c459bacdf9e4c8d278553` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.309492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b6d625d413c459bacdf9e4c8d278553)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:04.311579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.324931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 824/1693] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:04 2022] PUT /identity/v3/projects/15b0d1bf85e0404282622a3a3213d2c4/users/7cdead2514334b62afa46f4e61f14e96/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:04.329916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.330322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.330744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.378612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:04.379915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:04.471875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 825/1694] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:04.475439 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.479615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:04.500650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ed64bb-5a96-4f0c-b408-152d7d232dd6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.507207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.508795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.510795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.511156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] PATH_INFO: `/v3/projects/f6d845930c7a4c39af048eff19871f04` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.512646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6d845930c7a4c39af048eff19871f04)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:04.514880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.596596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b6d625d413c459bacdf9e4c8d278553'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.606228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b6d625d413c459bacdf9e4c8d278553'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.607249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b6d625d413c459bacdf9e4c8d278553'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.607998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b6d625d413c459bacdf9e4c8d278553'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.635815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6d845930c7a4c39af048eff19871f04', 'request_id': 'req-0b76eb43-d386-4657-9b46-73fd09f5bddc'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.691477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b6d625d413c459bacdf9e4c8d278553'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.744721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 870/1695] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:04 2022] DELETE /identity/v3/users/8b6d625d413c459bacdf9e4c8d278553 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:04.746160 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.751794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:04.770643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6d845930c7a4c39af048eff19871f04', 'request_id': 'req-0b76eb43-d386-4657-9b46-73fd09f5bddc'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.793701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6d845930c7a4c39af048eff19871f04', 'request_id': 'req-0b76eb43-d386-4657-9b46-73fd09f5bddc'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.834545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:04.842701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] The token cache is being invalidate because project f6d845930c7a4c39af048eff19871f04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:04.843026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.845231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-0b76eb43-d386-4657-9b46-73fd09f5bddc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:04.847984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 826/1696] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:04 2022] DELETE /identity/v3/projects/f6d845930c7a4c39af048eff19871f04 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:04.849705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463a42a7-fed6-4e33-961c-99787f126fdb None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.849993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463a42a7-fed6-4e33-961c-99787f126fdb None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.850457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463a42a7-fed6-4e33-961c-99787f126fdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.906680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-463a42a7-fed6-4e33-961c-99787f126fdb None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:04.907918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-463a42a7-fed6-4e33-961c-99787f126fdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:04.914842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720ce76-e0be-491a-9f94-5c589c8da914 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.928646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b08ed6-7dae-4f8c-93fd-d60a0a4401f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.929255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b08ed6-7dae-4f8c-93fd-d60a0a4401f7 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.930922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b08ed6-7dae-4f8c-93fd-d60a0a4401f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.930922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b08ed6-7dae-4f8c-93fd-d60a0a4401f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.934752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-88b08ed6-7dae-4f8c-93fd-d60a0a4401f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:04.981788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:05.095557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 827/1697] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:05.096858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662c902b-f3a3-4c09-b17c-135dc73a6303 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.097141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662c902b-f3a3-4c09-b17c-135dc73a6303 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.097410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662c902b-f3a3-4c09-b17c-135dc73a6303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.156988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b08ed6-7dae-4f8c-93fd-d60a0a4401f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:05.158531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b08ed6-7dae-4f8c-93fd-d60a0a4401f7 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:05.169560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-662c902b-f3a3-4c09-b17c-135dc73a6303 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:05.170532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-662c902b-f3a3-4c09-b17c-135dc73a6303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:05.188443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 871/1698] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:05.199087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 828/1699] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:05.208975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.213128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.213128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.248188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:05.252498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:05.283139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662c902b-f3a3-4c09-b17c-135dc73a6303 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:05.290494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 872/1700] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:05.299705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662c902b-f3a3-4c09-b17c-135dc73a6303 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:05.311631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:05.312351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.312703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.313044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.313769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:05.314139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:05.315841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:05.322207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:05.323917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:05.350194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc74267-b80b-494b-a373-840a87a7206c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:05.350839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 829/1701] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:05.352489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:05.361871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:05.367715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.369561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.370012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.386168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1224e-aeab-4bb9-992b-4ccc1c17112e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:05.394391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6690f34-62e8-482e-9ce1-8525a4d6c0fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:05.395115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6690f34-62e8-482e-9ce1-8525a4d6c0fb admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.395460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6690f34-62e8-482e-9ce1-8525a4d6c0fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.395788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6690f34-62e8-482e-9ce1-8525a4d6c0fb admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.396554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6690f34-62e8-482e-9ce1-8525a4d6c0fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:05.402072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6690f34-62e8-482e-9ce1-8525a4d6c0fb admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:05.436053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:05.441602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:05.502023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 873/1702] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:05.549113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 830/1703] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:05 2022] POST /identity/v3/projects => generated 365 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:05.555934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:05.570927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d7e49-27c9-4214-996d-397697117002 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:05.701408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:05.702044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.702319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.702620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.705471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:05.708028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:05.767577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.767863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.768164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.816852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 874/1704] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:05 2022] POST /identity/v3/users => generated 283 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:05.821948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:05.839405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:05.844508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] MFA Rules not processed for user `835a297b6e75416ea9bd1a82b466b2a9`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:05.853169 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:05.853316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4b1267-22dc-4fca-b448-07d88d2a6c5c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:05.864613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:05.864790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.865068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.865336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.865987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:05.871521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:05.899354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 875/1705] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:05 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:05.900714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:05.926129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c1ca2-c9dd-4f45-8655-fd15db65a972 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:05.935315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:05.940270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:05.940648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:05.940923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:05.941619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:05.943299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:05.970135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 876/1706] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:05 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:05.970711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:05.993692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a244c57b-c397-46d4-a87c-c28b23152e1a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:06.004498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 831/1707] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:06.006643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b080a2-9e37-4047-aa2b-3392f5de8ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:06.006794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b080a2-9e37-4047-aa2b-3392f5de8ed0 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:06.007041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b080a2-9e37-4047-aa2b-3392f5de8ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:06.007473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b080a2-9e37-4047-aa2b-3392f5de8ed0 admin admin] PATH_INFO: `/v3/projects/9aab2ddd2ddb4deda0d7aa7360c552cf/users/3d42c64eb2444582a759fa44b4a500f9/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:06.019685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:06.022086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:06.023748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:06.045396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b080a2-9e37-4047-aa2b-3392f5de8ed0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9aab2ddd2ddb4deda0d7aa7360c552cf, user_id=3d42c64eb2444582a759fa44b4a500f9, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:06.049287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b080a2-9e37-4047-aa2b-3392f5de8ed0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:06.150293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 877/1708] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:05 2022] PUT /identity/v3/projects/9aab2ddd2ddb4deda0d7aa7360c552cf/users/3d42c64eb2444582a759fa44b4a500f9/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:06.160735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c89f6c-fdd5-476b-bea9-f4cd169a6dfa None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:06.161333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c89f6c-fdd5-476b-bea9-f4cd169a6dfa None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:06.162012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c89f6c-fdd5-476b-bea9-f4cd169a6dfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:06.187911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fe587-6a07-4fa7-86d7-f577e92891e0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:06.229318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-14c89f6c-fdd5-476b-bea9-f4cd169a6dfa None None] MFA Rules not processed for user `3d42c64eb2444582a759fa44b4a500f9`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:06.231432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-14c89f6c-fdd5-476b-bea9-f4cd169a6dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:06.330996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:06.331804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:06.333010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:06.333386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:06.356885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 878/1709] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:06 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:06.357865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c89f6c-fdd5-476b-bea9-f4cd169a6dfa None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:06.551257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:06.553392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:06.556689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:06.562898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c89f6c-fdd5-476b-bea9-f4cd169a6dfa None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:06.576478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:06.579982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:06.580393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:06.582680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 832/1710] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:06 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 564 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:06.582895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:06.583982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:06.586639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:06.660016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:06.660016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:06.677264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a658b42c-db38-4c57-afca-1f39d83463bd service nova] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:06.684286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:06.685307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:06.685848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:06.686321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] PATH_INFO: `/v3/projects/14861bfe355b443c9a3be140e2990a80` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:06.687974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14861bfe355b443c9a3be140e2990a80)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:06.691060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:06.692558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 879/1711] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:06 2022] POST /identity/v3/projects => generated 363 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:06.697248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:06.905189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14861bfe355b443c9a3be140e2990a80', 'request_id': 'req-51eabf0f-d92c-43da-bff6-4afcb2e6c332'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:06.973069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca72ccb2-1bb8-4cf0-ae98-3413820610fb admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:07.055935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14861bfe355b443c9a3be140e2990a80', 'request_id': 'req-51eabf0f-d92c-43da-bff6-4afcb2e6c332'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.063379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:07.064321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:07.064481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:07.064715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:07.065927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:07.068052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:07.069269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14861bfe355b443c9a3be140e2990a80', 'request_id': 'req-51eabf0f-d92c-43da-bff6-4afcb2e6c332'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.114608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] The token cache is being invalidate because project 14861bfe355b443c9a3be140e2990a80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:07.116224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.122069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.129601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 833/1712] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:06 2022] DELETE /identity/v3/projects/14861bfe355b443c9a3be140e2990a80 => generated 0 bytes in 476 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:07.135842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:07.139645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:07.174985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51eabf0f-d92c-43da-bff6-4afcb2e6c332 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:07.197191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:07.199547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:07.199846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:07.202584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] PATH_INFO: `/v3/users/a7c7b5c7f46048f5bc1cc4fa245d636a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:07.206997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 880/1713] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:06 2022] POST /identity/v3/users => generated 288 bytes in 509 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:07.209027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7c7b5c7f46048f5bc1cc4fa245d636a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:07.209765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:07.212613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:07.215138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:07.263522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970e9ec-9af8-4b36-be1f-ff7b070c6bda admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:07.279077 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:07.282258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:07.282947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:07.282947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:07.283540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:07.285183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:07.307185 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 881/1714] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:07 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:07.318509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:07.352409 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c6b252-3a27-4327-9f75-703a992d5731 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:07.363223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:07.364167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:07.370230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:07.370230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] PATH_INFO: `/v3/projects/3016424e4c83420682dbcb9aca98d4eb/users/bf729a2d0e8f4a43b20608202232687a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:07.394935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3016424e4c83420682dbcb9aca98d4eb, user_id=bf729a2d0e8f4a43b20608202232687a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:07.399530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:07.463615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7c7b5c7f46048f5bc1cc4fa245d636a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.490348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7c7b5c7f46048f5bc1cc4fa245d636a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.491329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7c7b5c7f46048f5bc1cc4fa245d636a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.492716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7c7b5c7f46048f5bc1cc4fa245d636a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.517083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:07.554293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 882/1715] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:07 2022] PUT /identity/v3/projects/3016424e4c83420682dbcb9aca98d4eb/users/bf729a2d0e8f4a43b20608202232687a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:07.560795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:07.565363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:07.607132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7c7b5c7f46048f5bc1cc4fa245d636a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:07.714170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 834/1716] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:07 2022] DELETE /identity/v3/users/a7c7b5c7f46048f5bc1cc4fa245d636a => generated 0 bytes in 580 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:07.720252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:07.722220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:07.837295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae1f09b-c920-47cc-9b0d-23352b06c60a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:07.896829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39929a9b-1bd3-4626-a38c-15c033159438 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:07.918806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be75a08-a581-4e53-8bad-e36fe38bb172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:07.919570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be75a08-a581-4e53-8bad-e36fe38bb172 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:07.919950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be75a08-a581-4e53-8bad-e36fe38bb172 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:07.920502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be75a08-a581-4e53-8bad-e36fe38bb172 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:07.924164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be75a08-a581-4e53-8bad-e36fe38bb172 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:07.925949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be75a08-a581-4e53-8bad-e36fe38bb172 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:07.958994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 835/1717] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:07 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:07.960628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f75299-eedc-4254-ae81-fc5b9f1cbac4 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:07.961110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f75299-eedc-4254-ae81-fc5b9f1cbac4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:07.961684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f75299-eedc-4254-ae81-fc5b9f1cbac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:07.992484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:07.993455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:07.993829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:07.994202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] PATH_INFO: `/v3/projects/cb42f476994e409aa5cd54dbc9331051` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:07.995883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb42f476994e409aa5cd54dbc9331051)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:07.998788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:08.040421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-05f75299-eedc-4254-ae81-fc5b9f1cbac4 None None] MFA Rules not processed for user `7cdead2514334b62afa46f4e61f14e96`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:08.041901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-05f75299-eedc-4254-ae81-fc5b9f1cbac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:08.178848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb42f476994e409aa5cd54dbc9331051', 'request_id': 'req-9c6b332f-8262-411d-b450-9d149d9be8da'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:08.202886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb42f476994e409aa5cd54dbc9331051', 'request_id': 'req-9c6b332f-8262-411d-b450-9d149d9be8da'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:08.211946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:08.239797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 836/1718] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:07 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:08.240895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f75299-eedc-4254-ae81-fc5b9f1cbac4 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:08.267503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f75299-eedc-4254-ae81-fc5b9f1cbac4 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:08.296273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb42f476994e409aa5cd54dbc9331051', 'request_id': 'req-9c6b332f-8262-411d-b450-9d149d9be8da'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:08.316363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] The token cache is being invalidate because project cb42f476994e409aa5cd54dbc9331051 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:08.316948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:08.321496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:08.335456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 883/1719] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:07 2022] DELETE /identity/v3/projects/cb42f476994e409aa5cd54dbc9331051 => generated 0 bytes in 777 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:08.335456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:08.341969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:08.440065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:08.440065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:08.440383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:08.440451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] PATH_INFO: `/v3/projects/3016424e4c83420682dbcb9aca98d4eb/users/bf729a2d0e8f4a43b20608202232687a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:08.442584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3016424e4c83420682dbcb9aca98d4eb, user_id=bf729a2d0e8f4a43b20608202232687a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:08.449600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:08.492064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 837/1720] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:08 2022] PUT /identity/v3/projects/3016424e4c83420682dbcb9aca98d4eb/users/bf729a2d0e8f4a43b20608202232687a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:08.492064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:08.494544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:08.520402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6b332f-8262-411d-b450-9d149d9be8da admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:08.664596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b0a479-7ed5-439a-9f0e-36ac7c815c47 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:08.672878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:08.673791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:08.674311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:08.674829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] PATH_INFO: `/v3/users/6b06a1bc014f41838b794b6f34db76a5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:08.686787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b06a1bc014f41838b794b6f34db76a5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:08.689601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:08.715472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35977166-9ca8-49e1-852d-b91b93ef02b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:08.718120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35977166-9ca8-49e1-852d-b91b93ef02b1 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:08.719636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35977166-9ca8-49e1-852d-b91b93ef02b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:08.721159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35977166-9ca8-49e1-852d-b91b93ef02b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:08.722758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35977166-9ca8-49e1-852d-b91b93ef02b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:08.728228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35977166-9ca8-49e1-852d-b91b93ef02b1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:08.853377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 884/1721] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:08 2022] POST /identity/v3/projects => generated 403 bytes in 521 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:08.857192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eafee9-efd1-4f11-91cc-f24e9c853061 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:08.858747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eafee9-efd1-4f11-91cc-f24e9c853061 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:08.859316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eafee9-efd1-4f11-91cc-f24e9c853061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:08.910769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-25eafee9-efd1-4f11-91cc-f24e9c853061 None None] MFA Rules not processed for user `bf729a2d0e8f4a43b20608202232687a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:08.912215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-25eafee9-efd1-4f11-91cc-f24e9c853061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:08.918139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:09.070944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b06a1bc014f41838b794b6f34db76a5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:09.082233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 885/1722] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:08 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:09.083800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eafee9-efd1-4f11-91cc-f24e9c853061 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:09.098046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b06a1bc014f41838b794b6f34db76a5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:09.106951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b06a1bc014f41838b794b6f34db76a5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:09.108353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b06a1bc014f41838b794b6f34db76a5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:09.223008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eafee9-efd1-4f11-91cc-f24e9c853061 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:09.228488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b06a1bc014f41838b794b6f34db76a5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:09.238351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:09.239017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:09.239294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:09.239564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:09.240197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:09.243071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:09.259438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:09.266517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 838/1723] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:08 2022] DELETE /identity/v3/users/6b06a1bc014f41838b794b6f34db76a5 => generated 0 bytes in 782 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:09.267754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:09.270502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:09.402233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed914d4-8e02-4bf5-b05c-ca32d0cd9b0f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:09.403832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 886/1724] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:09 2022] POST /identity/v3/users => generated 308 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:09.407691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:09.411418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:09.416445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:09.420981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:09.421939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:09.422366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:09.427183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:09.462713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f4f830-6ae1-4152-b8d7-091d15bd10a3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:09.587599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:09.589513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:09.621359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:09.622008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:09.622290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:09.622637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:09.628024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:09.629183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 839/1725] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:09 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:09.629712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:09.658627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 887/1726] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:09 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:09.666185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:09.681042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32741a97-31f5-426f-8424-e48fe438ade6 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:09.695310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:09.697417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:09.697705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:09.700538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] PATH_INFO: `/v3/projects/36968ac344f244dea75243d5b6e716b6/users/aa8af11279874354b0ee9af198201932/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:09.778252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] RBAC: Authorizing `identity:create_grant(project_id=36968ac344f244dea75243d5b6e716b6, user_id=aa8af11279874354b0ee9af198201932, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:09.781468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:09.878161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:09.878694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 840/1727] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:09 2022] PUT /identity/v3/projects/36968ac344f244dea75243d5b6e716b6/users/aa8af11279874354b0ee9af198201932/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:09.905955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446b900-b3d2-4ad1-989c-c87c72a42adf admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:10.035992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:10.040308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:10.040308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:10.040308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:10.040308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:10.045110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:10.080315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:10.080799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 888/1728] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:09 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:10.094706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053a9495-e85b-454a-a53b-419aa3d6248a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:10.104604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e375fc2e-59bc-4518-8609-d2c5e42a3a67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:10.106365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e375fc2e-59bc-4518-8609-d2c5e42a3a67 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:10.106884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e375fc2e-59bc-4518-8609-d2c5e42a3a67 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:10.107424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e375fc2e-59bc-4518-8609-d2c5e42a3a67 admin admin] PATH_INFO: `/v3/projects/36968ac344f244dea75243d5b6e716b6/users/aa8af11279874354b0ee9af198201932/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:10.110531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e375fc2e-59bc-4518-8609-d2c5e42a3a67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36968ac344f244dea75243d5b6e716b6, user_id=aa8af11279874354b0ee9af198201932, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:10.117666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e375fc2e-59bc-4518-8609-d2c5e42a3a67 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:10.137597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 841/1729] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:10 2022] PUT /identity/v3/projects/36968ac344f244dea75243d5b6e716b6/users/aa8af11279874354b0ee9af198201932/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:10.185018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:10.188022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:10.252648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eb4a07-c6d1-44e1-a01c-e211aec73005 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:10.304961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcfe68-9b17-4e21-85e4-24a620eecbf4 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:10.304961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcfe68-9b17-4e21-85e4-24a620eecbf4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:10.305269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcfe68-9b17-4e21-85e4-24a620eecbf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:10.374702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:10.375386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:10.375640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:10.375917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] PATH_INFO: `/v3/projects/ccdd7ea571124358b17262fec27f5cf2` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:10.387299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9dbcfe68-9b17-4e21-85e4-24a620eecbf4 None None] MFA Rules not processed for user `1dcf55e151b94d379bcf21073f574af5`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:10.388601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ccdd7ea571124358b17262fec27f5cf2)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:10.389623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9dbcfe68-9b17-4e21-85e4-24a620eecbf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:10.412433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:10.416260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:10.578465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 842/1730] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:10.583042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea6704c-1e9c-4ad4-92bb-9741abba1c48 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:10.583531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea6704c-1e9c-4ad4-92bb-9741abba1c48 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:10.583814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea6704c-1e9c-4ad4-92bb-9741abba1c48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:10.608483 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ccdd7ea571124358b17262fec27f5cf2', 'request_id': 'req-21b51e10-5650-4828-a64b-100ca012d4a3'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:10.634479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ccdd7ea571124358b17262fec27f5cf2', 'request_id': 'req-21b51e10-5650-4828-a64b-100ca012d4a3'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:10.644521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7ea6704c-1e9c-4ad4-92bb-9741abba1c48 None None] MFA Rules not processed for user `1dcf55e151b94d379bcf21073f574af5`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:10.649198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7ea6704c-1e9c-4ad4-92bb-9741abba1c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:10.736171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ccdd7ea571124358b17262fec27f5cf2', 'request_id': 'req-21b51e10-5650-4828-a64b-100ca012d4a3'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:10.786678 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] The token cache is being invalidate because project ccdd7ea571124358b17262fec27f5cf2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:10.786678 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:10.788244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:10.810953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 889/1731] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:10 2022] DELETE /identity/v3/projects/ccdd7ea571124358b17262fec27f5cf2 => generated 0 bytes in 614 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:10.813332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 843/1732] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:10.815020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:10.815434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:10.815978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:10.822568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:10.827554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:10.939326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:10.940724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:10.998631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:11.102064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b51e10-5650-4828-a64b-100ca012d4a3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:11.136727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:11.138866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:11.141057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:11.147164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] PATH_INFO: `/v3/projects/8d00871c71bd40e48425cb00d85c0d90` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:11.153034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d00871c71bd40e48425cb00d85c0d90)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:11.155048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:11.250340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 844/1733] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:11.253638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:11.258764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:11.363630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d00871c71bd40e48425cb00d85c0d90', 'request_id': 'req-12ee3e52-4c86-4f8d-b3c9-7b662e113992'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:11.393517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d00871c71bd40e48425cb00d85c0d90', 'request_id': 'req-12ee3e52-4c86-4f8d-b3c9-7b662e113992'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:11.464557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d00871c71bd40e48425cb00d85c0d90', 'request_id': 'req-12ee3e52-4c86-4f8d-b3c9-7b662e113992'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:11.467877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61281ace-2444-4529-a2bc-0a01c4ad5db3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:11.490236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] The token cache is being invalidate because project 8d00871c71bd40e48425cb00d85c0d90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:11.492788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:11.499267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:11.501252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 890/1734] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:10 2022] DELETE /identity/v3/projects/8d00871c71bd40e48425cb00d85c0d90 => generated 0 bytes in 690 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:11.502337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:11.506307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:11.644494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ca1e7c-62e1-4d96-a538-d4a86902f9b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:11.645141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ca1e7c-62e1-4d96-a538-d4a86902f9b7 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:11.654199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ca1e7c-62e1-4d96-a538-d4a86902f9b7 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:11.654612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ca1e7c-62e1-4d96-a538-d4a86902f9b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:11.658111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e0ca1e7c-62e1-4d96-a538-d4a86902f9b7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:11.746177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee3e52-4c86-4f8d-b3c9-7b662e113992 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:11.760863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:11.762439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:11.764647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:11.766880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] PATH_INFO: `/v3/users/3d42c64eb2444582a759fa44b4a500f9` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:11.767479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d42c64eb2444582a759fa44b4a500f9)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:11.773590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:11.860102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ca1e7c-62e1-4d96-a538-d4a86902f9b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:11.864685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ca1e7c-62e1-4d96-a538-d4a86902f9b7 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:11.918871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 845/1735] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:11 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 668 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:11.920402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e093e89d-fc69-4171-a92d-920a77ce3141 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:11.920677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e093e89d-fc69-4171-a92d-920a77ce3141 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:11.920949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e093e89d-fc69-4171-a92d-920a77ce3141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:11.999263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e093e89d-fc69-4171-a92d-920a77ce3141 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:12.000527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e093e89d-fc69-4171-a92d-920a77ce3141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:12.026203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 846/1736] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:12.031875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:12.032364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:12.032771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:12.066270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d42c64eb2444582a759fa44b4a500f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:12.094639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d42c64eb2444582a759fa44b4a500f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:12.099162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d42c64eb2444582a759fa44b4a500f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:12.103829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:12.105317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d42c64eb2444582a759fa44b4a500f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:12.119536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] MFA Rules not processed for user `185e359e571f44bda1585124529b36a8`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:12.120804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:12.208977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1a025674-9069-4f7a-bcbd-6d26985ba831 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d42c64eb2444582a759fa44b4a500f9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:12.310293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 891/1737] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:11 2022] DELETE /identity/v3/users/3d42c64eb2444582a759fa44b4a500f9 => generated 0 bytes in 805 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:12.314566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923346f1-d263-4a30-b2d9-2b8054a1ad0d None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:12.315824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923346f1-d263-4a30-b2d9-2b8054a1ad0d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:12.316262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923346f1-d263-4a30-b2d9-2b8054a1ad0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:12.345137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-923346f1-d263-4a30-b2d9-2b8054a1ad0d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:12.347467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-923346f1-d263-4a30-b2d9-2b8054a1ad0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:12.349612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 847/1738] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:12.353649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:12.360668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:12.393386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:12.395531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee611bb1-ff41-470a-add8-67eeaa35e85c None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:12.511899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 892/1739] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:12.514769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9623ce1-2ccd-451e-9c47-63813ffae169 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:12.515921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9623ce1-2ccd-451e-9c47-63813ffae169 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:12.516210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9623ce1-2ccd-451e-9c47-63813ffae169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:12.574658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf0a3e-431c-43d2-9db0-f605e050a218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:12.580695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf0a3e-431c-43d2-9db0-f605e050a218 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:12.583774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf0a3e-431c-43d2-9db0-f605e050a218 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:12.583774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf0a3e-431c-43d2-9db0-f605e050a218 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:12.583774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf0a3e-431c-43d2-9db0-f605e050a218 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:12.583774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf0a3e-431c-43d2-9db0-f605e050a218 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:12.588307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf0a3e-431c-43d2-9db0-f605e050a218 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:12.589461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e9623ce1-2ccd-451e-9c47-63813ffae169 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:12.592538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e9623ce1-2ccd-451e-9c47-63813ffae169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:12.641911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 893/1740] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:12.644834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:12.645180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:12.645456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:12.654063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 848/1741] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:12.658804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c40399-4faf-468c-ab2c-390c8142b192 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:12.659041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c40399-4faf-468c-ab2c-390c8142b192 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:12.659301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c40399-4faf-468c-ab2c-390c8142b192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:12.748706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] MFA Rules not processed for user `185e359e571f44bda1585124529b36a8`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:12.756564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-14c40399-4faf-468c-ab2c-390c8142b192 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:12.760511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-14c40399-4faf-468c-ab2c-390c8142b192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:12.762623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:12.794919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 849/1742] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:12.795621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c40399-4faf-468c-ab2c-390c8142b192 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:12.817768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c40399-4faf-468c-ab2c-390c8142b192 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:12.828652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b100166c-01d8-4df0-917a-4dc0965e7562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:12.830065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b100166c-01d8-4df0-917a-4dc0965e7562 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:12.830554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b100166c-01d8-4df0-917a-4dc0965e7562 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:12.831137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b100166c-01d8-4df0-917a-4dc0965e7562 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:12.848238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b100166c-01d8-4df0-917a-4dc0965e7562 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:12.850728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b100166c-01d8-4df0-917a-4dc0965e7562 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:12.913102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 894/1743] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:12 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:12.914158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:12.924979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:13.005410 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 850/1744] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:12 2022] POST /identity/v3/projects => generated 387 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:13.006973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e347f92b-a234-4444-aafb-aec94f50d129 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.010450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e347f92b-a234-4444-aafb-aec94f50d129 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.010450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e347f92b-a234-4444-aafb-aec94f50d129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.071815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e347f92b-a234-4444-aafb-aec94f50d129 None None] MFA Rules not processed for user `14d56042415540a3b3ed06578b712ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:13.073799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e347f92b-a234-4444-aafb-aec94f50d129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:13.097812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738e39e2-e3d6-4d04-a5ce-638748bb851a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:13.164469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:13.170980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 851/1745] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:13 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:13.172040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e347f92b-a234-4444-aafb-aec94f50d129 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:13.199795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:13.200432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.200834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.201119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.277874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:13.279563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:13.292737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 895/1746] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:13.293794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:13.295606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:13.332061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e347f92b-a234-4444-aafb-aec94f50d129 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:13.356245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:13.356876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.362337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.362671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.363380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:13.363695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:13.370066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:13.378523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:DELETE=1,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:13.397662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 852/1747] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:13.398630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:13.419847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb6138c-ddbf-4153-978f-181ad721d46c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:13.443656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c66962d-25df-48c7-9356-523afab2f7d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:13.444286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c66962d-25df-48c7-9356-523afab2f7d5 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.444563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c66962d-25df-48c7-9356-523afab2f7d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.445545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c66962d-25df-48c7-9356-523afab2f7d5 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.446762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c66962d-25df-48c7-9356-523afab2f7d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:13.449322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c66962d-25df-48c7-9356-523afab2f7d5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:13.454573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd40933-0fc4-41f8-98a9-50aeb3615947 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:13.463954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a51247-2a24-4a77-9ea7-7dd88bdad777 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:13.465170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a51247-2a24-4a77-9ea7-7dd88bdad777 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.465170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a51247-2a24-4a77-9ea7-7dd88bdad777 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.465170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a51247-2a24-4a77-9ea7-7dd88bdad777 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.545010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a51247-2a24-4a77-9ea7-7dd88bdad777 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:13.549110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a51247-2a24-4a77-9ea7-7dd88bdad777 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:13.555282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 853/1748] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:13 2022] POST /identity/v3/users => generated 294 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:13.556870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d9e87-5c0a-4618-93e8-d6d3c04e8ddf None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.557516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d9e87-5c0a-4618-93e8-d6d3c04e8ddf None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.557807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d9e87-5c0a-4618-93e8-d6d3c04e8ddf None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.560324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 854/1749] 10.222.0.27 () {54 vars in 912 bytes} [Tue Oct 4 08:52:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:52:13.561853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d9e87-5c0a-4618-93e8-d6d3c04e8ddf None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:13.590304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 896/1750] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:13 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:13.592060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d9e87-5c0a-4618-93e8-d6d3c04e8ddf None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:13.592684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4307e134-88f0-4543-882d-c5bb70fdded9 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.592976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4307e134-88f0-4543-882d-c5bb70fdded9 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.594857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4307e134-88f0-4543-882d-c5bb70fdded9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.608918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:13.609657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.610056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.610209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.610880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:13.619847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:13.715239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-4307e134-88f0-4543-882d-c5bb70fdded9 None None] MFA Rules not processed for user `aa8af11279874354b0ee9af198201932`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:13.716646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4307e134-88f0-4543-882d-c5bb70fdded9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:13.733656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 855/1751] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:13 2022] POST /identity/v3/projects => generated 381 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:13.735153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:13.899766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 897/1752] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:13 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:13.901304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.901649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.902520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.970055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:13.971331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2db3b07-ca24-4fdb-8402-c5b26fdaae9b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:13.985306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:13.986092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:13.987312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:13.987734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:13.988534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:13.995013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.014570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] MFA Rules not processed for user `df391d2bf7884538823f04db5564d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:14.017912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 856/1753] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:13 2022] GET /identity/v3/roles => generated 1420 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:14.020147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.022432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:14.045110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ed7072-8d67-4b3a-acd4-b1b2e962fa45 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.056272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.056910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.057592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.067468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.067468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.067468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.134942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 898/1754] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:52:13 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:14.136146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.138579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:14.162184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480b0d37-c755-41d7-9fb6-6d67c7d0ae4f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.176734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.177623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.178043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.178517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] PATH_INFO: `/v3/projects/9aab2ddd2ddb4deda0d7aa7360c552cf` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.185356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] RBAC: Authorizing `identity:delete_project(project_id=9aab2ddd2ddb4deda0d7aa7360c552cf)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.187445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.238616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 857/1755] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:14 2022] POST /identity/v3/users => generated 291 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:14.239531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.253798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8f3c69-373e-48d6-bf95-51ecf1ef86bb admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.259041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.259665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.260144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.260496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.261156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.262834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.278586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 858/1756] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:14.278586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.290896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6cd3f9-f745-4d1e-b64f-ce4ef7b4a31a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.294723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c05b38-1717-47ff-be6a-738fa99b2a63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.295516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c05b38-1717-47ff-be6a-738fa99b2a63 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.295778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c05b38-1717-47ff-be6a-738fa99b2a63 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.296051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c05b38-1717-47ff-be6a-738fa99b2a63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.305247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c05b38-1717-47ff-be6a-738fa99b2a63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.307206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c05b38-1717-47ff-be6a-738fa99b2a63 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.316418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 859/1757] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:14.317761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20c688-0362-43fb-b93f-b1544de7ad2e None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.324115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20c688-0362-43fb-b93f-b1544de7ad2e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.324115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20c688-0362-43fb-b93f-b1544de7ad2e None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.324115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 860/1758] 10.222.0.27 () {54 vars in 912 bytes} [Tue Oct 4 08:52:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:52:14.328110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a20c688-0362-43fb-b93f-b1544de7ad2e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.343501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9aab2ddd2ddb4deda0d7aa7360c552cf', 'request_id': 'req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:14.351537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9aab2ddd2ddb4deda0d7aa7360c552cf', 'request_id': 'req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:14.431323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a20c688-0362-43fb-b93f-b1544de7ad2e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.437920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b210d3d-65b0-44f7-90f2-5a0888cef263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.438915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b210d3d-65b0-44f7-90f2-5a0888cef263 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.439333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b210d3d-65b0-44f7-90f2-5a0888cef263 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.439740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b210d3d-65b0-44f7-90f2-5a0888cef263 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.440817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b210d3d-65b0-44f7-90f2-5a0888cef263 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.443704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9aab2ddd2ddb4deda0d7aa7360c552cf', 'request_id': 'req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:14.452499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b210d3d-65b0-44f7-90f2-5a0888cef263 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.461472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] The token cache is being invalidate because project 9aab2ddd2ddb4deda0d7aa7360c552cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:14.462777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:14.466230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:14.470276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 899/1759] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:14 2022] DELETE /identity/v3/projects/9aab2ddd2ddb4deda0d7aa7360c552cf => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:14.473849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.488638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 861/1760] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:14 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:14.488819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:14.489778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.489778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.495026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.511042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8448c9a-235d-4e27-b0bd-f2fe68c64eef admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.517445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.518501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.519061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.519481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] PATH_INFO: `/v3/projects/28804e2dfe4843a9986332f3ca9b9323/users/54a97a8259a04817b2a89f73ad13b48e/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.537043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] MFA Rules not processed for user `df391d2bf7884538823f04db5564d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:14.539506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:14.545097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28804e2dfe4843a9986332f3ca9b9323, user_id=54a97a8259a04817b2a89f73ad13b48e, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.549199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.625324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 862/1761] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:52:14 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:14.626697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.628958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:14.647517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e4d7ac-708a-42ac-ba48-01ca07cf0eb6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.655787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:14.656040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 900/1762] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:14 2022] PUT /identity/v3/projects/28804e2dfe4843a9986332f3ca9b9323/users/54a97a8259a04817b2a89f73ad13b48e/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:14.657095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.666914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:14.753869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.755372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.755963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.756465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] PATH_INFO: `/v3/users/bf729a2d0e8f4a43b20608202232687a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.759376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf729a2d0e8f4a43b20608202232687a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.765393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.802183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c279e3-b541-4d21-81d3-3d4c98205063 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.811876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.812747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.813255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.813771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.814624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.818800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.846560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 901/1763] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:14 2022] GET /identity/v3/roles => generated 1420 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:14.846560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:14.847972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:14.939885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c46b4-9afc-4d0e-9af5-b39e6726fecf admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:14.943967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed06fdaf-87bc-4bb3-92b4-fef533497192 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:14.945580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed06fdaf-87bc-4bb3-92b4-fef533497192 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.945693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed06fdaf-87bc-4bb3-92b4-fef533497192 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.945955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed06fdaf-87bc-4bb3-92b4-fef533497192 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:14.962061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed06fdaf-87bc-4bb3-92b4-fef533497192 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:14.965022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed06fdaf-87bc-4bb3-92b4-fef533497192 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:14.985726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 902/1764] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:14.987511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:14.988017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:14.988482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:15.021859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf729a2d0e8f4a43b20608202232687a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:15.033552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf729a2d0e8f4a43b20608202232687a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:15.034461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf729a2d0e8f4a43b20608202232687a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:15.035426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf729a2d0e8f4a43b20608202232687a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:15.041384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] MFA Rules not processed for user `54a97a8259a04817b2a89f73ad13b48e`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:15.042893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.129058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf729a2d0e8f4a43b20608202232687a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:15.166449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 863/1765] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:14 2022] DELETE /identity/v3/users/bf729a2d0e8f4a43b20608202232687a => generated 0 bytes in 541 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:15.170451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 903/1766] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:14 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:15.170451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:15.171691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:15.173177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.173291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.228242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:15.319203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2109b571-88ec-4f96-a90f-f053952ddbbc admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:15.320277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37434a31-7435-479f-868f-b408b96ce0b1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:15.327651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:15.328360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:15.328641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:15.329095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] PATH_INFO: `/v3/projects/cfa63aec061c4a9d8a02ffa80677bf2e/users/d43b6ed9dabb428cad04d9be67fe522d/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:15.357927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfa63aec061c4a9d8a02ffa80677bf2e, user_id=d43b6ed9dabb428cad04d9be67fe522d, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:15.362365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:15.445577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 864/1767] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:15 2022] PUT /identity/v3/projects/cfa63aec061c4a9d8a02ffa80677bf2e/users/d43b6ed9dabb428cad04d9be67fe522d/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:15.448423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:15.457822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.477383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:15.477383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:15.477383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:15.477383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:15.484240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.500655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:15.640022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:15.649944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be238af-712f-4259-88cd-3e530048f46e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:15.661655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e044cd-348d-4847-aa48-63aeee3e58a2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:15.662322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e044cd-348d-4847-aa48-63aeee3e58a2 service placement] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:15.664202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e044cd-348d-4847-aa48-63aeee3e58a2 service placement] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:15.664490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e044cd-348d-4847-aa48-63aeee3e58a2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:15.668161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f9e044cd-348d-4847-aa48-63aeee3e58a2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.681331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:15.683128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:15.720122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e044cd-348d-4847-aa48-63aeee3e58a2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:15.722595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e044cd-348d-4847-aa48-63aeee3e58a2 service placement] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:15.742435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 904/1768] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:15 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 574 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:15.744122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:15.746126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.764934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 865/1769] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:15.768327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acf45d8-612f-4bd3-b75e-99497b47fe6c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:15.769171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acf45d8-612f-4bd3-b75e-99497b47fe6c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:15.769457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acf45d8-612f-4bd3-b75e-99497b47fe6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:15.802939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a313a7b-a483-414b-a705-e8f88cc890b4 service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:15.845611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-2acf45d8-612f-4bd3-b75e-99497b47fe6c None None] MFA Rules not processed for user `d43b6ed9dabb428cad04d9be67fe522d`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:15.882322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2acf45d8-612f-4bd3-b75e-99497b47fe6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:15.949018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa5be8a-9f52-4fe7-9fc5-e4d4022dc334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:15.950811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5be8a-9f52-4fe7-9fc5-e4d4022dc334 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:15.957369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5be8a-9f52-4fe7-9fc5-e4d4022dc334 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:15.958064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5be8a-9f52-4fe7-9fc5-e4d4022dc334 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:15.959011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa5be8a-9f52-4fe7-9fc5-e4d4022dc334 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:15.965609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa5be8a-9f52-4fe7-9fc5-e4d4022dc334 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:16.121576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 905/1770] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:15 2022] POST /identity/v3/projects => generated 389 bytes in 378 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:16.124122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849693e6-19d7-4125-ba0d-02e1b1e40c08 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:16.129130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849693e6-19d7-4125-ba0d-02e1b1e40c08 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:16.131314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849693e6-19d7-4125-ba0d-02e1b1e40c08 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:16.138243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 906/1771] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:52:16 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:52:16.144672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b496be-ba59-4798-a706-b42bc2d6faff None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:16.144672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b496be-ba59-4798-a706-b42bc2d6faff None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:16.145322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b496be-ba59-4798-a706-b42bc2d6faff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:16.187845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 866/1772] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 419 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:16.187845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acf45d8-612f-4bd3-b75e-99497b47fe6c None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:16.207765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-99b496be-ba59-4798-a706-b42bc2d6faff None None] MFA Rules not processed for user `835a297b6e75416ea9bd1a82b466b2a9`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:16.210327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-99b496be-ba59-4798-a706-b42bc2d6faff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:16.216913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acf45d8-612f-4bd3-b75e-99497b47fe6c None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:16.296056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 907/1773] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:16 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:16.302664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b496be-ba59-4798-a706-b42bc2d6faff None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:16.410581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:16.435141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef8b333-2130-4933-aee8-4fc23a00fb51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:16.435783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef8b333-2130-4933-aee8-4fc23a00fb51 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:16.436065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef8b333-2130-4933-aee8-4fc23a00fb51 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:16.436341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef8b333-2130-4933-aee8-4fc23a00fb51 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:16.443733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef8b333-2130-4933-aee8-4fc23a00fb51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:16.447867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef8b333-2130-4933-aee8-4fc23a00fb51 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:16.568247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 867/1774] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:16 2022] POST /identity/v3/users => generated 301 bytes in 381 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:16.568247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea4063-270e-48aa-a3cf-97146c59aee9 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:16.568247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea4063-270e-48aa-a3cf-97146c59aee9 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:16.568247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea4063-270e-48aa-a3cf-97146c59aee9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:16.582674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b496be-ba59-4798-a706-b42bc2d6faff None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:16.592041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa37e93-c99d-490a-b627-3f4d117f65ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:16.593699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa37e93-c99d-490a-b627-3f4d117f65ec admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:16.593985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa37e93-c99d-490a-b627-3f4d117f65ec admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:16.596499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa37e93-c99d-490a-b627-3f4d117f65ec admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:16.597337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa37e93-c99d-490a-b627-3f4d117f65ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:16.601286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa37e93-c99d-490a-b627-3f4d117f65ec admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:16.632936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-fcea4063-270e-48aa-a3cf-97146c59aee9 None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:16.636884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fcea4063-270e-48aa-a3cf-97146c59aee9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:16.689975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:16.746232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 908/1775] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:16 2022] POST /identity/v3/projects => generated 379 bytes in 449 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:16.752057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5337b2-b188-4d33-9fef-3da2cf73961c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:16.752351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5337b2-b188-4d33-9fef-3da2cf73961c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:16.753844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5337b2-b188-4d33-9fef-3da2cf73961c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:16.800950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 868/1776] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:52:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:16.802178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea4063-270e-48aa-a3cf-97146c59aee9 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:16.805571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1d5337b2-b188-4d33-9fef-3da2cf73961c None None] MFA Rules not processed for user `185e359e571f44bda1585124529b36a8`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:16.811197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1d5337b2-b188-4d33-9fef-3da2cf73961c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:16.830739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea4063-270e-48aa-a3cf-97146c59aee9 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:17.021079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 909/1777] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:16 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:17.021701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5337b2-b188-4d33-9fef-3da2cf73961c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:17.040293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:17.041194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.041517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.042059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.044626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:17.047223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:17.072436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 869/1778] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:16 2022] GET /identity/v3/roles => generated 1420 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:17.075220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:17.080027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:17.105986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a68f7ec-903b-495b-a444-916fe15a6f5f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:17.119868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:17.120651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.121033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.121566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] PATH_INFO: `/v3/projects/3016424e4c83420682dbcb9aca98d4eb` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.123219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3016424e4c83420682dbcb9aca98d4eb)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:17.125240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:17.245943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5337b2-b188-4d33-9fef-3da2cf73961c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:17.256080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:17.256968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.258054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.259868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.260681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:17.268007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:17.329174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3016424e4c83420682dbcb9aca98d4eb', 'request_id': 'req-7bf65e9f-9c2a-497a-a055-e5eb51eca674'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:17.392646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 910/1779] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:17 2022] POST /identity/v3/users => generated 296 bytes in 367 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:17.392646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:17.423011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c16d62-e63d-4760-920a-8682286260fd admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:17.508520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3016424e4c83420682dbcb9aca98d4eb', 'request_id': 'req-7bf65e9f-9c2a-497a-a055-e5eb51eca674'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:17.518488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3016424e4c83420682dbcb9aca98d4eb', 'request_id': 'req-7bf65e9f-9c2a-497a-a055-e5eb51eca674'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:17.538112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:17.546177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] The token cache is being invalidate because project 3016424e4c83420682dbcb9aca98d4eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:17.546379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1891a2da-187e-4ec7-9a67-c43fb2eaed5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:17.546505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:17.547435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1891a2da-187e-4ec7-9a67-c43fb2eaed5f admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.547860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1891a2da-187e-4ec7-9a67-c43fb2eaed5f admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.548270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1891a2da-187e-4ec7-9a67-c43fb2eaed5f admin admin] PATH_INFO: `/v3/projects/a702898b65df4652b997d8deed822006/users/b5c844a6965e4ca392cb688583edf868/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.550571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7bf65e9f-9c2a-497a-a055-e5eb51eca674 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:17.558312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 870/1780] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:17 2022] DELETE /identity/v3/projects/3016424e4c83420682dbcb9aca98d4eb => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:17.559751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce9f0a-d419-4d4f-93b0-3a6f5617cc95 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.560037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce9f0a-d419-4d4f-93b0-3a6f5617cc95 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.560312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce9f0a-d419-4d4f-93b0-3a6f5617cc95 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.561633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 871/1781] 10.222.0.27 () {54 vars in 961 bytes} [Tue Oct 4 08:52:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:52:17.571742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ce9f0a-d419-4d4f-93b0-3a6f5617cc95 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:17.576242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1891a2da-187e-4ec7-9a67-c43fb2eaed5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a702898b65df4652b997d8deed822006, user_id=b5c844a6965e4ca392cb688583edf868, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:17.579450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-12ce9f0a-d419-4d4f-93b0-3a6f5617cc95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:17.580066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1891a2da-187e-4ec7-9a67-c43fb2eaed5f admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:17.621253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:17.629414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ce9f0a-d419-4d4f-93b0-3a6f5617cc95 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:17.644828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54de1f-9367-4ed2-9c76-d08d92ff3a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:17.645491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54de1f-9367-4ed2-9c76-d08d92ff3a60 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.645769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54de1f-9367-4ed2-9c76-d08d92ff3a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.646039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54de1f-9367-4ed2-9c76-d08d92ff3a60 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.646738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc54de1f-9367-4ed2-9c76-d08d92ff3a60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:17.652734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc54de1f-9367-4ed2-9c76-d08d92ff3a60 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:17.683672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 872/1782] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:17 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:17.685313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96a4a1-0ba0-49e9-91b7-10ef219f6855 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.685606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96a4a1-0ba0-49e9-91b7-10ef219f6855 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.685882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96a4a1-0ba0-49e9-91b7-10ef219f6855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.689275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 911/1783] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:17 2022] PUT /identity/v3/projects/a702898b65df4652b997d8deed822006/users/b5c844a6965e4ca392cb688583edf868/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:17.695752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0104debf-7969-4c86-810f-0016773c3345 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.696036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0104debf-7969-4c86-810f-0016773c3345 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.696303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0104debf-7969-4c86-810f-0016773c3345 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.724992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-4a96a4a1-0ba0-49e9-91b7-10ef219f6855 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:17.727130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4a96a4a1-0ba0-49e9-91b7-10ef219f6855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:17.750191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0104debf-7969-4c86-810f-0016773c3345 None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:17.754202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0104debf-7969-4c86-810f-0016773c3345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:17.900603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 873/1784] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:17.901815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96a4a1-0ba0-49e9-91b7-10ef219f6855 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:17.916669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96a4a1-0ba0-49e9-91b7-10ef219f6855 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:17.924980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:17.926015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.926625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.927350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] PATH_INFO: `/v3/projects/f6d898b976d64df8977ac7cbc48c4245/users/6b749dbe9c434d0694d4e18fe35ed800/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.932837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 912/1785] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Oct 4 08:52:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:17.933515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0104debf-7969-4c86-810f-0016773c3345 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:17.938938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0104debf-7969-4c86-810f-0016773c3345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:17.958929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6d898b976d64df8977ac7cbc48c4245, user_id=6b749dbe9c434d0694d4e18fe35ed800, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:17.965021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:17.967406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0104debf-7969-4c86-810f-0016773c3345 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:17.978955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433a46ed-ce5c-4c7e-9002-a0ed2682b64a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:17.979593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a46ed-ce5c-4c7e-9002-a0ed2682b64a admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:17.979874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a46ed-ce5c-4c7e-9002-a0ed2682b64a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:17.980149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a46ed-ce5c-4c7e-9002-a0ed2682b64a admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:17.980921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433a46ed-ce5c-4c7e-9002-a0ed2682b64a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:17.985516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433a46ed-ce5c-4c7e-9002-a0ed2682b64a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:18.014453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 913/1786] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:17 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:18.018386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce2bd0-2d56-4ab8-afdf-5e2e3fb89a3e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.019010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce2bd0-2d56-4ab8-afdf-5e2e3fb89a3e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.019286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce2bd0-2d56-4ab8-afdf-5e2e3fb89a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.054615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f9ce2bd0-2d56-4ab8-afdf-5e2e3fb89a3e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:18.057051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f9ce2bd0-2d56-4ab8-afdf-5e2e3fb89a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:18.067367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 914/1787] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:18.073275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce2bd0-2d56-4ab8-afdf-5e2e3fb89a3e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:18.100951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 874/1788] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:17 2022] PUT /identity/v3/projects/f6d898b976d64df8977ac7cbc48c4245/users/6b749dbe9c434d0694d4e18fe35ed800/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:18.103141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:18.117529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce2bd0-2d56-4ab8-afdf-5e2e3fb89a3e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:18.188271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:18.240332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a028cd-a1e5-4e85-9682-1b82a5e91067 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:18.246721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:18.247415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.247700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.247977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] PATH_INFO: `/v3/projects/a702898b65df4652b997d8deed822006/users/b5c844a6965e4ca392cb688583edf868/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.250256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a702898b65df4652b997d8deed822006, user_id=b5c844a6965e4ca392cb688583edf868, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:18.252974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:18.279346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 875/1789] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:18 2022] PUT /identity/v3/projects/a702898b65df4652b997d8deed822006/users/b5c844a6965e4ca392cb688583edf868/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:18.281269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:18.292804 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:18.293402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.293686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.293960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.295882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:18.321064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:18.323432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:18.343071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 915/1790] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:18.344480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:18.369493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af57194-2229-4d34-a656-9163a79f7ab9 service placement] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:18.433392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dc0e31-6423-4bdb-86bc-3cde903ad295 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:18.444046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:18.446814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.447548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.449206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.450868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:18.451651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:18.461152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:18.495989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 876/1791] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:18.496984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:18.500612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:18.515874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de98bee8-ae92-4238-9c61-6f038966e4b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:18.521188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eea465-9220-43bd-8d64-05ecbd7ea930 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:18.522096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:18.523473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.523793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.524126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] PATH_INFO: `/v3/users/7cdead2514334b62afa46f4e61f14e96` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.525371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7cdead2514334b62afa46f4e61f14e96)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:18.527285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de98bee8-ae92-4238-9c61-6f038966e4b2 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.527534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:18.529298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de98bee8-ae92-4238-9c61-6f038966e4b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.529937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de98bee8-ae92-4238-9c61-6f038966e4b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.532821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de98bee8-ae92-4238-9c61-6f038966e4b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:18.534828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de98bee8-ae92-4238-9c61-6f038966e4b2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:18.546020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 916/1792] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:18 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:18.548389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ae8f7-49d3-4ae0-b6e9-b9f3e35a281b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.548958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ae8f7-49d3-4ae0-b6e9-b9f3e35a281b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.549512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7ae8f7-49d3-4ae0-b6e9-b9f3e35a281b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.611908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-4d7ae8f7-49d3-4ae0-b6e9-b9f3e35a281b None None] MFA Rules not processed for user `b5c844a6965e4ca392cb688583edf868`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:18.614992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4d7ae8f7-49d3-4ae0-b6e9-b9f3e35a281b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:18.776000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7cdead2514334b62afa46f4e61f14e96'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:18.791499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7cdead2514334b62afa46f4e61f14e96'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:18.792437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7cdead2514334b62afa46f4e61f14e96'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:18.793043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7cdead2514334b62afa46f4e61f14e96'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:18.865332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 917/1793] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:18 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:18.868719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7ae8f7-49d3-4ae0-b6e9-b9f3e35a281b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:18.891552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7ae8f7-49d3-4ae0-b6e9-b9f3e35a281b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:18.895384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7cdead2514334b62afa46f4e61f14e96'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:18.905684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:18.906632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:18.907054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:18.907454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:18.908255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:18.911136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:18.949568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 877/1794] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:18 2022] DELETE /identity/v3/users/7cdead2514334b62afa46f4e61f14e96 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:18.970911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:18.974108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:19.038987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 918/1795] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:18 2022] POST /identity/v3/projects => generated 385 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:19.040647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:19.043823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:19.254442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57dbb55-5b61-4283-8920-978548cff52d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:19.276788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c2558b-eef7-4e37-b344-2724187ba66f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:19.277685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c30b8f1-807e-4cf2-b0e6-17d16ce6785f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:19.280090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c30b8f1-807e-4cf2-b0e6-17d16ce6785f service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:19.280638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c30b8f1-807e-4cf2-b0e6-17d16ce6785f service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:19.286063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c30b8f1-807e-4cf2-b0e6-17d16ce6785f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:19.286063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0c30b8f1-807e-4cf2-b0e6-17d16ce6785f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:19.385672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:19.386285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:19.386587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:19.386862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] PATH_INFO: `/v3/projects/f6d898b976d64df8977ac7cbc48c4245/users/6b749dbe9c434d0694d4e18fe35ed800/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:19.390285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6d898b976d64df8977ac7cbc48c4245, user_id=6b749dbe9c434d0694d4e18fe35ed800, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:19.396926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:19.418780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 878/1796] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:18 2022] PUT /identity/v3/projects/f6d898b976d64df8977ac7cbc48c4245/users/6b749dbe9c434d0694d4e18fe35ed800/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:19.419870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:19.423616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:19.427850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c30b8f1-807e-4cf2-b0e6-17d16ce6785f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:19.429938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c30b8f1-807e-4cf2-b0e6-17d16ce6785f service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:19.441776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 919/1797] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:19 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:19.443343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:19.443689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:19.445364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:19.523692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] MFA Rules not processed for user `6b749dbe9c434d0694d4e18fe35ed800`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:19.525190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:19.532221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:19.556942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080c1ef3-501f-4e7c-b07c-56a5061e76e7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:19.567327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:19.571000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:19.571208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:19.571440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:19.573641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:19.588244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:19.693356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 920/1798] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:19 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:19.694178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:19.696258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:19.715706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 879/1799] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:19 2022] POST /identity/v3/users => generated 293 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:19.717053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:19.719180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:19.724182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3a8f08-aa7f-423e-ad1d-d0ef7f942d1e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:19.732013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:19.733727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9885748d-dc73-40d2-9748-789d47287c8f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:19.734101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:19.734470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:19.734712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] PATH_INFO: `/v3/projects/15b0d1bf85e0404282622a3a3213d2c4` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:19.736155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=15b0d1bf85e0404282622a3a3213d2c4)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:19.738243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:19.739567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:19.740515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:19.740916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:19.741203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] PATH_INFO: `/v3/users/54a97a8259a04817b2a89f73ad13b48e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:19.749919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] RBAC: Authorizing `identity:delete_user(user_id=54a97a8259a04817b2a89f73ad13b48e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:19.754350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:19.958244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:20.016551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54a97a8259a04817b2a89f73ad13b48e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.016779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15b0d1bf85e0404282622a3a3213d2c4', 'request_id': 'req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.030224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54a97a8259a04817b2a89f73ad13b48e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.033484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54a97a8259a04817b2a89f73ad13b48e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.033593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15b0d1bf85e0404282622a3a3213d2c4', 'request_id': 'req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.033686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54a97a8259a04817b2a89f73ad13b48e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.112954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54a97a8259a04817b2a89f73ad13b48e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.133395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 880/1800] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:19 2022] DELETE /identity/v3/users/54a97a8259a04817b2a89f73ad13b48e => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:20.134261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:20.140889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:20.208329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15b0d1bf85e0404282622a3a3213d2c4', 'request_id': 'req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.224776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] The token cache is being invalidate because project 15b0d1bf85e0404282622a3a3213d2c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:20.225080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.226666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:20.229124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 921/1801] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:19 2022] DELETE /identity/v3/projects/15b0d1bf85e0404282622a3a3213d2c4 => generated 0 bytes in 535 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:20.235012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:20.238860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:20.277981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88febb4c-486d-4577-b7b5-da6bb843b460 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:20.281367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d73d314-2d02-4c21-a48e-090d2abfee15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:20.282289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d73d314-2d02-4c21-a48e-090d2abfee15 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:20.283185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d73d314-2d02-4c21-a48e-090d2abfee15 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:20.283882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d73d314-2d02-4c21-a48e-090d2abfee15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:20.286320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1d73d314-2d02-4c21-a48e-090d2abfee15 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:20.376919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f2f7d0-b5fa-4ec7-8d1d-9d35f7f1df3a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:20.448422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d73d314-2d02-4c21-a48e-090d2abfee15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:20.451242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d73d314-2d02-4c21-a48e-090d2abfee15 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:20.475335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 881/1802] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:20 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:20.481266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a162f0d-6cff-4e8a-a938-d24cc7b9c294 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:20.481897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a162f0d-6cff-4e8a-a938-d24cc7b9c294 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:20.482424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a162f0d-6cff-4e8a-a938-d24cc7b9c294 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:20.556473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0a162f0d-6cff-4e8a-a938-d24cc7b9c294 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:20.558717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0a162f0d-6cff-4e8a-a938-d24cc7b9c294 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:20.589240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:20.589940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:20.597530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:20.598110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:20.600887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:20.602156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:20.611473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:20.650881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 922/1803] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:20 2022] GET /identity/v3/roles => generated 1420 bytes in 421 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:20.652716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:20.660498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:20.672202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:20.701781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b24f16-e8d9-4ba8-94ce-6fc1f22a3256 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:20.716463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:20.717130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:20.717412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:20.717710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] PATH_INFO: `/v3/users/aa8af11279874354b0ee9af198201932` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:20.719203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa8af11279874354b0ee9af198201932)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:20.721699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:20.828325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 882/1804] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:20.829359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a162f0d-6cff-4e8a-a938-d24cc7b9c294 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:20.884143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a162f0d-6cff-4e8a-a938-d24cc7b9c294 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:20.901268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:20.901964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:20.906763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:20.907060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:20.908542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:20.911184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:20.950600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 883/1805] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:20 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:20.952160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:20.956210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:21.005522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa8af11279874354b0ee9af198201932'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:21.021522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa8af11279874354b0ee9af198201932'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:21.023664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa8af11279874354b0ee9af198201932'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:21.025103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa8af11279874354b0ee9af198201932'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:21.111428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7d41c-94df-46bc-9e46-7df58af46a07 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:21.116025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:21.116888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:21.117555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:21.117934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:21.120803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:21.129408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8606c437-7f6c-45a4-92ca-7479b2f91494 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa8af11279874354b0ee9af198201932'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:21.136553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:21.138549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:21.151507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 884/1806] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:21.152992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:21.154475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:21.170080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1676d40a-36bf-4e17-bf43-5bc8d7bc9b07 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:21.170232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 923/1807] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:20 2022] DELETE /identity/v3/users/aa8af11279874354b0ee9af198201932 => generated 0 bytes in 519 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:21.171542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02e2e5-b3e4-4515-8933-54448dadd9fe None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:21.173188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02e2e5-b3e4-4515-8933-54448dadd9fe None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:21.173577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02e2e5-b3e4-4515-8933-54448dadd9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:21.210935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-2e02e2e5-b3e4-4515-8933-54448dadd9fe None None] MFA Rules not processed for user `14d56042415540a3b3ed06578b712ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:21.212775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2e02e2e5-b3e4-4515-8933-54448dadd9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:21.247443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:21.248466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:21.248882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:21.249353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] PATH_INFO: `/v3/projects/21aeed44c97e4bad8ac40eef9dd2774f/users/d6312463243a4bfd9fd489d7d31886a6/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:21.293950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=21aeed44c97e4bad8ac40eef9dd2774f, user_id=d6312463243a4bfd9fd489d7d31886a6, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:21.297043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:21.358608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 924/1808] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:21.418309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 885/1809] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:21 2022] PUT /identity/v3/projects/21aeed44c97e4bad8ac40eef9dd2774f/users/d6312463243a4bfd9fd489d7d31886a6/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:21.428536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:21.428773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:21.429068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:21.473292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] MFA Rules not processed for user `d6312463243a4bfd9fd489d7d31886a6`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:21.474910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:21.565954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 925/1810] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:21.570372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:21.659828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:21.660744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:21.736466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e3ee5-f61f-43d8-abdd-1916006c51ca admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:21.755997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:21.773224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fe3ece-b213-4ea8-91d4-68d0e2ec90e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:21.774208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe3ece-b213-4ea8-91d4-68d0e2ec90e6 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:21.776712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe3ece-b213-4ea8-91d4-68d0e2ec90e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:21.777042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe3ece-b213-4ea8-91d4-68d0e2ec90e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:21.778383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fe3ece-b213-4ea8-91d4-68d0e2ec90e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:21.791291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fe3ece-b213-4ea8-91d4-68d0e2ec90e6 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:21.883897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93468a52-c5ea-4cf6-8ae6-757f2d49a80b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:21.911378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:21.918665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:21.919000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:21.919245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] PATH_INFO: `/v3/projects/36968ac344f244dea75243d5b6e716b6` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:21.924792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36968ac344f244dea75243d5b6e716b6)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:21.926804 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:21.939986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 886/1811] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:21 2022] POST /identity/v3/projects => generated 387 bytes in 370 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:21.941749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:21.942712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:21.943792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.064423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:22.065160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:22.152590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36968ac344f244dea75243d5b6e716b6', 'request_id': 'req-6f483270-89c8-48c9-9ffb-5bf890074f00'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.161249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36968ac344f244dea75243d5b6e716b6', 'request_id': 'req-6f483270-89c8-48c9-9ffb-5bf890074f00'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.244312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36968ac344f244dea75243d5b6e716b6', 'request_id': 'req-6f483270-89c8-48c9-9ffb-5bf890074f00'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.265119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] The token cache is being invalidate because project 36968ac344f244dea75243d5b6e716b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:22.265489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.266223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.269706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 926/1812] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:21 2022] DELETE /identity/v3/projects/36968ac344f244dea75243d5b6e716b6 => generated 0 bytes in 611 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:22.270225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:22.272974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:22.344645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 887/1813] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:52:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 404 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:22.345422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:22.348030 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:22.434777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f483270-89c8-48c9-9ffb-5bf890074f00 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:22.461116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc47c29a-236a-489d-be76-2c050ee4d834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:22.461759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc47c29a-236a-489d-be76-2c050ee4d834 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:22.462186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc47c29a-236a-489d-be76-2c050ee4d834 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:22.462489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc47c29a-236a-489d-be76-2c050ee4d834 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.463497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc47c29a-236a-489d-be76-2c050ee4d834 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:22.468788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc47c29a-236a-489d-be76-2c050ee4d834 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:22.477475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e49e6d4-4771-4df7-8b43-f2606b6eecba None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:22.513028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:22.580073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 927/1814] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:22 2022] POST /identity/v3/users => generated 300 bytes in 307 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:22.580073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:22.580073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:22.580073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.621075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:22.629599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:22.635968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6b749dbe9c434d0694d4e18fe35ed800', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6d898b976d64df8977ac7cbc48c4245', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:22.637216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:22.637678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:22.638160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.639037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:22.642107 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:22.650251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 928/1815] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:22.650251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:22.650251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:22.655693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:22.661930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c22cf8-fae1-4fe5-93a4-34478bb7b5bc None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:22.682831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:22.684824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:22.685219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:22.685372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] PATH_INFO: `/v3/projects/28804e2dfe4843a9986332f3ca9b9323` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.688978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28804e2dfe4843a9986332f3ca9b9323)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:22.692353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 888/1816] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:22.693422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:22.698245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:22.726550 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e606d0-0cf7-41ba-8e27-0ee2450ef311 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:22.742804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5e652-7128-4ade-942b-8d6e569b1c25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:22.743383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5e652-7128-4ade-942b-8d6e569b1c25 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:22.743732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5e652-7128-4ade-942b-8d6e569b1c25 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:22.744080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5e652-7128-4ade-942b-8d6e569b1c25 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.746039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5e652-7128-4ade-942b-8d6e569b1c25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:22.751194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e5e652-7128-4ade-942b-8d6e569b1c25 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:22.765301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 889/1817] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:22 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:22.767148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c87320-96e0-44e2-8c36-942ee6417db2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:22.771060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c87320-96e0-44e2-8c36-942ee6417db2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:22.771060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c87320-96e0-44e2-8c36-942ee6417db2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.809893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-51c87320-96e0-44e2-8c36-942ee6417db2 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:22.811376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-51c87320-96e0-44e2-8c36-942ee6417db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:22.821280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 890/1818] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:22.822199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c87320-96e0-44e2-8c36-942ee6417db2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:22.837381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c87320-96e0-44e2-8c36-942ee6417db2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:22.844243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6b749dbe9c434d0694d4e18fe35ed800', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6d898b976d64df8977ac7cbc48c4245', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:22.845149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:22.845682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:22.846012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:22.846952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:22.849227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:22.855749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:22.891627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28804e2dfe4843a9986332f3ca9b9323', 'request_id': 'req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.900935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28804e2dfe4843a9986332f3ca9b9323', 'request_id': 'req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.939368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 891/1819] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:22 2022] POST /identity/v3/projects => generated 354 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:22.939368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:22.974858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28804e2dfe4843a9986332f3ca9b9323', 'request_id': 'req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.996352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] The token cache is being invalidate because project 28804e2dfe4843a9986332f3ca9b9323 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:22.996731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:22.997521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:23.005574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 929/1820] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:22 2022] DELETE /identity/v3/projects/28804e2dfe4843a9986332f3ca9b9323 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:23.007551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:23.012339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:23.105467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6a893a-5817-45ef-a9df-79022b8ee267 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:23.110713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:23.111397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:23.111675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:23.112063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] PATH_INFO: `/v3/projects/9583aadcd4274a3e8f9b73d0c3b36910/users/61abe27d9517434bb026799cbff73a34/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:23.129418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9583aadcd4274a3e8f9b73d0c3b36910, user_id=61abe27d9517434bb026799cbff73a34, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:23.132698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:23.179777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d5ab4-b7f6-4870-9d06-35af4602b4e7 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:23.188588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:23.191202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:23.191489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:23.191768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:23.192442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:23.192750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:23.199061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:23.211783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 892/1821] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:22 2022] PUT /identity/v3/projects/9583aadcd4274a3e8f9b73d0c3b36910/users/61abe27d9517434bb026799cbff73a34/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:23.211783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:23.213157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:23.229062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 930/1822] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:23.239058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:23.244291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:23.401816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83473bf3-0f6f-481d-a262-dcece6cb771e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:23.409231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:23.410154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:23.410868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:23.411357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:23.414715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:23.417767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c281c205-e7ab-4574-822b-9d6a3a54f93d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:23.502014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:23.541039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6b749dbe9c434d0694d4e18fe35ed800', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6d898b976d64df8977ac7cbc48c4245', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:23.542124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:23.542450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:23.542769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:23.543490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:23.543800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:23.546214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:23.552429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:23.554950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:23.566825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 893/1823] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:52:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:52:23.573549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:23.573549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:23.586735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 931/1824] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:23.588077 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:23.591209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:23.741407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e5527c-63f4-43d6-a200-192a81cea66f service swift] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:23.758994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:23.759636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:23.760876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:23.761163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] PATH_INFO: `/v3/users/b5c844a6965e4ca392cb688583edf868` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:23.763573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5c844a6965e4ca392cb688583edf868)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:23.766455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:23.767791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb83545-76e1-45bc-b1ad-a30bb71d62dc tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:23.782530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:23.785744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:23.786081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:23.786535 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:23.787281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:23.792203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:23.822566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 932/1825] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:23 2022] GET /identity/v3/roles => generated 1420 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:23.822566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:23.860060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af78a10-b7db-4ad6-8352-ea297fdf3f5e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:23.899309 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:23.899958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:23.900241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:23.900600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:23.901987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:23.911798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:24.099467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 933/1826] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:23 2022] POST /identity/v3/projects => generated 365 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:24.100279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:24.118871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:24.176210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5c844a6965e4ca392cb688583edf868'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.229079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5c844a6965e4ca392cb688583edf868'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.233854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5c844a6965e4ca392cb688583edf868'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.236485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5c844a6965e4ca392cb688583edf868'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.315295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b35c52-ada5-4816-b92d-c00eff395c4b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:24.315295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6b749dbe9c434d0694d4e18fe35ed800', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6d898b976d64df8977ac7cbc48c4245', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:24.329241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:24.330463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:24.330788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] PATH_INFO: `/v3/projects/0909d069e0694fcda00440d5ae85fd9c` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:24.335744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorizing `identity:delete_project(project_id=0909d069e0694fcda00440d5ae85fd9c)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:24.343127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:24.598611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5c844a6965e4ca392cb688583edf868'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.608562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 894/1827] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:23 2022] DELETE /identity/v3/users/b5c844a6965e4ca392cb688583edf868 => generated 0 bytes in 1041 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:24.611127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:24.614950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:24.705698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802a40b5-2275-42d1-b661-6d4d08da811e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:24.712195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:24.712954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:24.713284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:24.713562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] PATH_INFO: `/v3/projects/9583aadcd4274a3e8f9b73d0c3b36910/users/61abe27d9517434bb026799cbff73a34/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:24.716102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9583aadcd4274a3e8f9b73d0c3b36910, user_id=61abe27d9517434bb026799cbff73a34, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:24.732254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:24.757527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 895/1828] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:24 2022] PUT /identity/v3/projects/9583aadcd4274a3e8f9b73d0c3b36910/users/61abe27d9517434bb026799cbff73a34/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:24.759661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:24.767143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:24.831772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0909d069e0694fcda00440d5ae85fd9c', 'request_id': 'req-fb83f35b-f9c9-4aad-a958-ff5226b6af84'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.859227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0909d069e0694fcda00440d5ae85fd9c', 'request_id': 'req-fb83f35b-f9c9-4aad-a958-ff5226b6af84'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.913589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:24.975175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0909d069e0694fcda00440d5ae85fd9c', 'request_id': 'req-fb83f35b-f9c9-4aad-a958-ff5226b6af84'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.993235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] The token cache is being invalidate because project 0909d069e0694fcda00440d5ae85fd9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:24.993460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb801550-ccb4-4542-af82-9eead9fbb64b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:24.993693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:24.995100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb83f35b-f9c9-4aad-a958-ff5226b6af84 tempest-VolumeQuotaClassesTest-632408546 tempest-VolumeQuotaClassesTest-632408546-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:25.000508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 934/1829] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:24 2022] DELETE /identity/v3/projects/0909d069e0694fcda00440d5ae85fd9c => generated 0 bytes in 900 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:25.001419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.001744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.002028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.088628 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] MFA Rules not processed for user `61abe27d9517434bb026799cbff73a34`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:25.091143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:25.162530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f1702a-8aeb-4de7-bc78-a88d5718a1e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.164337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f1702a-8aeb-4de7-bc78-a88d5718a1e7 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.167128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f1702a-8aeb-4de7-bc78-a88d5718a1e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.167533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f1702a-8aeb-4de7-bc78-a88d5718a1e7 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.168304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f1702a-8aeb-4de7-bc78-a88d5718a1e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.171039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f1702a-8aeb-4de7-bc78-a88d5718a1e7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:25.295991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 896/1830] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:24 2022] POST /identity/v3/users => generated 283 bytes in 534 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:25.297505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.297798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.298077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.387421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] MFA Rules not processed for user `df391d2bf7884538823f04db5564d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:25.395510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:25.401488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 935/1831] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:25 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 401 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:25.402545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:25.421569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:25.464135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da38cf69-3625-447e-acf0-bfa9339b9332 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:25.484807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 897/1832] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Oct 4 08:52:25 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:25.486048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:25.487797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:25.503807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.504576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.507368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.508229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.510745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.516049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:25.554222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 936/1833] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:25 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:25.555742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:25.563234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:25.626095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e01d970-36ea-4c30-ac95-f1286e74378e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:25.628065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5691962-b29f-4bb0-9881-5b401c506a05 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:25.635149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.637199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.650644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.651054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.651304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.652403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.654939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.655244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.677929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.680459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:25.698218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 937/1834] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:25.700024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:25.708359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.710525 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:25.748889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4253f13e-3d74-46a6-a934-6ac754380a83 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:25.757017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 898/1835] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:25 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:25.758190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:25.759544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:25.764183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:25.784099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.784099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.784099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.784099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.784698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.784698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:25.809206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a145ced2-c54b-4060-80c2-1f697bc21412 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:25.810695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.811494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.811927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.812566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.848027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.855635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 938/1836] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:25 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:25.859672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:25.869717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:25.873600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 899/1837] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:25.901723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:25.905547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac37f36-a0f8-4e53-a7ed-3118d9c65f02 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:25.920775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:25.922814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:25.932607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6568a664-7174-48ab-9562-a1798f2f2a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.933769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568a664-7174-48ab-9562-a1798f2f2a45 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.934247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568a664-7174-48ab-9562-a1798f2f2a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.934816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568a664-7174-48ab-9562-a1798f2f2a45 admin admin] PATH_INFO: `/v3/projects/9733341358a3481388a52c4724ed053d/users/2878f48cff82499ab41c2237b3afdaa2/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.951638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3771a1ab-3e1b-4117-b9eb-ab211f06b20b service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:25.958284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6568a664-7174-48ab-9562-a1798f2f2a45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9733341358a3481388a52c4724ed053d, user_id=2878f48cff82499ab41c2237b3afdaa2, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.962434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:25.963231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:25.963524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:25.963800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] PATH_INFO: `/v3/users/d6312463243a4bfd9fd489d7d31886a6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:25.964433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6568a664-7174-48ab-9562-a1798f2f2a45 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:25.965273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6312463243a4bfd9fd489d7d31886a6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:25.967598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:26.085919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 939/1838] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:25 2022] PUT /identity/v3/projects/9733341358a3481388a52c4724ed053d/users/2878f48cff82499ab41c2237b3afdaa2/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:26.104607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:26.104900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:26.105865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:26.169274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] MFA Rules not processed for user `2878f48cff82499ab41c2237b3afdaa2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:26.174510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:26.267797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6312463243a4bfd9fd489d7d31886a6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:26.284252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6312463243a4bfd9fd489d7d31886a6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:26.289338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6312463243a4bfd9fd489d7d31886a6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:26.290206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6312463243a4bfd9fd489d7d31886a6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:26.404586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a6a39d62-b4e6-4499-9097-afaa6568c7f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6312463243a4bfd9fd489d7d31886a6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:26.435987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 940/1839] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:26.437142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:26.441158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:26.491936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 900/1840] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:25 2022] DELETE /identity/v3/users/d6312463243a4bfd9fd489d7d31886a6 => generated 0 bytes in 573 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:26.496554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:26.497531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:26.498376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:26.547097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:26.548841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:26.626102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:26.651233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 901/1841] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:26.667520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005c5d76-0bcb-499d-9a7f-8f33d1235539 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:26.674044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:26.675351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:26.676029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:26.676536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:26.677884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:26.681286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:26.713236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:26.715881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:26.780341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 941/1842] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:26 2022] POST /identity/v3/projects => generated 365 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:52:26.783443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:26.790547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:26.793304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:26.884558 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75edd5db-e7b0-4d97-aa5c-8db948fe2edf None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:26.927810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d266f9-c0a9-4761-a75a-05b84d806aba admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:26.935925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:26.936729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:26.937081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:26.937359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] PATH_INFO: `/v3/users/d43b6ed9dabb428cad04d9be67fe522d` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:26.939059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d43b6ed9dabb428cad04d9be67fe522d)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:26.942052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:27.018781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:27.021856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:27.023325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:27.024751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:27.039801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:27.045887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:27.060242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 902/1843] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:27.061467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:27.073687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be91795e-7d41-44ca-8509-bac644d53c0f service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:27.085216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:27.085216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:27.085216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:27.085216 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:27.085530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:27.085530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:27.236157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 903/1844] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:27 2022] POST /identity/v3/users => generated 289 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:27.250755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:27.265976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30165c3-3946-44ba-8225-c45227452760 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:27.280764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:27.296773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:27.301382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:27.302434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:27.302434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:27.302771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:27.309985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:27.317109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd43b6ed9dabb428cad04d9be67fe522d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.330905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd43b6ed9dabb428cad04d9be67fe522d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.332495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd43b6ed9dabb428cad04d9be67fe522d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.334145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd43b6ed9dabb428cad04d9be67fe522d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.364040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 904/1845] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:27 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:27.367287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:27.370675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:27.427715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7795ae3d-3e74-4ddb-a944-5cec3a9f18c9 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:27.435791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:27.436570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:27.436884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:27.437277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] PATH_INFO: `/v3/projects/a702898b65df4652b997d8deed822006` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:27.438677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a702898b65df4652b997d8deed822006)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:27.441022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:27.496591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd43b6ed9dabb428cad04d9be67fe522d'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.530114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 942/1846] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:26 2022] DELETE /identity/v3/users/d43b6ed9dabb428cad04d9be67fe522d => generated 0 bytes in 747 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:27.531679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:27.533477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:27.621468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a702898b65df4652b997d8deed822006', 'request_id': 'req-fe768edb-f425-4a59-a9fe-c516be8a1720'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.692669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f7be0-1976-473f-aa26-6da0cfd90997 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:27.706414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a702898b65df4652b997d8deed822006', 'request_id': 'req-fe768edb-f425-4a59-a9fe-c516be8a1720'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.718881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a702898b65df4652b997d8deed822006', 'request_id': 'req-fe768edb-f425-4a59-a9fe-c516be8a1720'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.762366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] The token cache is being invalidate because project a702898b65df4652b997d8deed822006 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:27.762366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.781681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fe768edb-f425-4a59-a9fe-c516be8a1720 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:27.793279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 905/1847] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:27 2022] DELETE /identity/v3/projects/a702898b65df4652b997d8deed822006 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:27.794865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e56ab-0d10-4afd-ae06-fe7c42c66061 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:27.795157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e56ab-0d10-4afd-ae06-fe7c42c66061 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:27.795438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2e56ab-0d10-4afd-ae06-fe7c42c66061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:27.806131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:27.884164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-af2e56ab-0d10-4afd-ae06-fe7c42c66061 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:27.887836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-af2e56ab-0d10-4afd-ae06-fe7c42c66061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:27.934829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:27.935541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:27.935826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:27.936107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] PATH_INFO: `/v3/projects/98b8b827e137460b8f138df67bfb589a/users/52b3a51f2e2b474385cfdb519dea679c/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:27.940440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:28.019394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98b8b827e137460b8f138df67bfb589a, user_id=52b3a51f2e2b474385cfdb519dea679c, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:28.023395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:28.121068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 906/1848] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:28.122630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169205dd-4d7d-4113-acb6-0698e830f37a None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.122920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169205dd-4d7d-4113-acb6-0698e830f37a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.123197 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169205dd-4d7d-4113-acb6-0698e830f37a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.136686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 943/1849] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:27 2022] PUT /identity/v3/projects/98b8b827e137460b8f138df67bfb589a/users/52b3a51f2e2b474385cfdb519dea679c/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 602 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:28.137996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:28.147492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:28.200583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-169205dd-4d7d-4113-acb6-0698e830f37a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:28.207302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-169205dd-4d7d-4113-acb6-0698e830f37a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:28.302526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3e6fd9-24f0-4972-8b48-55df04b8ed68 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:28.318863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:28.320549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.321075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.324029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.324029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:28.324619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:28.346416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 944/1850] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:28 2022] GET /identity/v3/roles => generated 1420 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:28.357439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:28.357439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:28.473989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493e9dd2-05bd-4976-85a0-579d0267c639 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:28.492586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d109be-092e-4b1f-8e6d-e0092d4ae5d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:28.494734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d109be-092e-4b1f-8e6d-e0092d4ae5d0 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.496374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d109be-092e-4b1f-8e6d-e0092d4ae5d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.497297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d109be-092e-4b1f-8e6d-e0092d4ae5d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.510842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 907/1851] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:28.518707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169205dd-4d7d-4113-acb6-0698e830f37a None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:28.523910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d109be-092e-4b1f-8e6d-e0092d4ae5d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:28.527315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d109be-092e-4b1f-8e6d-e0092d4ae5d0 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:28.531466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169205dd-4d7d-4113-acb6-0698e830f37a None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:28.537927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6459af8-b02e-44d9-b56b-4c3291044b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:28.540754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6459af8-b02e-44d9-b56b-4c3291044b5c admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.541052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6459af8-b02e-44d9-b56b-4c3291044b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.541337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6459af8-b02e-44d9-b56b-4c3291044b5c admin admin] PATH_INFO: `/v3/projects/98b8b827e137460b8f138df67bfb589a/users/52b3a51f2e2b474385cfdb519dea679c/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.542815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 945/1852] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:28.543415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6459af8-b02e-44d9-b56b-4c3291044b5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=98b8b827e137460b8f138df67bfb589a, user_id=52b3a51f2e2b474385cfdb519dea679c, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:28.544625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa81147-6c48-45e1-871b-25c55721aa74 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.545160 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa81147-6c48-45e1-871b-25c55721aa74 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.545635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa81147-6c48-45e1-871b-25c55721aa74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.547194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6459af8-b02e-44d9-b56b-4c3291044b5c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:28.577522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 908/1853] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:28 2022] PUT /identity/v3/projects/98b8b827e137460b8f138df67bfb589a/users/52b3a51f2e2b474385cfdb519dea679c/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:28.580356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.581313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.581795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.608643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-3fa81147-6c48-45e1-871b-25c55721aa74 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:28.609735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3fa81147-6c48-45e1-871b-25c55721aa74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:28.618387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:28.628696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] MFA Rules not processed for user `52b3a51f2e2b474385cfdb519dea679c`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:28.631629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:28.750859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 909/1854] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:28 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:28.762989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:28.764757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:28.782563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 946/1855] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:28.787907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa81147-6c48-45e1-871b-25c55721aa74 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:28.799217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa81147-6c48-45e1-871b-25c55721aa74 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:28.805827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:28.807487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.808321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.808818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.809817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:28.810687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:28.817123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:28.853411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:28.856104 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 947/1856] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:28.857081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:28.881683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38c8d3f-d7b1-428f-8e10-665a6af3fd7b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:28.896890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:28.898511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:28.901173 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:28.901455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:28.902101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:28.905896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:28.994697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a685e2-ea55-4b2b-8a9a-9ecdb0a95116 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:28.999880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:29.000920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.001576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.001953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:29.017255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:29.019111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:29.045168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 948/1857] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:28 2022] POST /identity/v3/projects => generated 379 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:29.045168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:29.048131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:29.063094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 910/1858] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:28 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:29.064752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:29.097874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af80a2e2-066f-48d5-8805-1c389b2af376 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:29.237104 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410db5db-2e46-4947-8b23-bf8063d6c61a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:29.245270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:29.246366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.246683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.247725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] PATH_INFO: `/v3/projects/21aeed44c97e4bad8ac40eef9dd2774f` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:29.248936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21aeed44c97e4bad8ac40eef9dd2774f)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:29.257441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:29.276347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:29.276347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.276347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.276347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:29.280748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:29.283924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:29.424214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 911/1859] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:29 2022] POST /identity/v3/users => generated 290 bytes in 360 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:29.425894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:29.442020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21aeed44c97e4bad8ac40eef9dd2774f', 'request_id': 'req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:29.464736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21aeed44c97e4bad8ac40eef9dd2774f', 'request_id': 'req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:29.539114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65011b92-d54a-4cba-a115-df867f155d2b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:29.546193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21aeed44c97e4bad8ac40eef9dd2774f', 'request_id': 'req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:29.572162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] The token cache is being invalidate because project 21aeed44c97e4bad8ac40eef9dd2774f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:29.573241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:29.578970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:29.582706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 949/1860] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:29 2022] DELETE /identity/v3/projects/21aeed44c97e4bad8ac40eef9dd2774f => generated 0 bytes in 536 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:29.586398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:29.591599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:29.717690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:29.718701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.718982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.719583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:29.737195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:29.781300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f9444c-a7cc-4a0e-b24a-61d1b8900336 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:29.787811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2217d34-9c73-4c10-a946-713db54fef5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:29.796806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2217d34-9c73-4c10-a946-713db54fef5c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.797128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2217d34-9c73-4c10-a946-713db54fef5c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.797407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2217d34-9c73-4c10-a946-713db54fef5c admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:29.798109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2217d34-9c73-4c10-a946-713db54fef5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:29.804908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2217d34-9c73-4c10-a946-713db54fef5c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:29.843083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 950/1861] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:29 2022] GET /identity/v3/roles => generated 1420 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:29.845143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b348ab-ed0d-4438-be4d-074da4036b8d None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.848203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b348ab-ed0d-4438-be4d-074da4036b8d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.848203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b348ab-ed0d-4438-be4d-074da4036b8d None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:29.848203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 951/1862] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:52:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:52:29.865146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b348ab-ed0d-4438-be4d-074da4036b8d None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:29.869019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a4b348ab-ed0d-4438-be4d-074da4036b8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:29.906358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b348ab-ed0d-4438-be4d-074da4036b8d None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:29.915407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:29.916558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.917846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:29.920429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.921233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] PATH_INFO: `/v3/users/61abe27d9517434bb026799cbff73a34` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:29.924826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=61abe27d9517434bb026799cbff73a34)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:29.930946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:29.946856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:29.949232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:29.952141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:29.971072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 912/1863] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:29 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:29.972127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:29.987600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d71a82-9a14-4530-9a4d-e3ce58c85deb service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:29.996621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101de742-4bda-46f2-b440-f25738514f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:29.998127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101de742-4bda-46f2-b440-f25738514f54 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:29.998643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101de742-4bda-46f2-b440-f25738514f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:29.999439 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101de742-4bda-46f2-b440-f25738514f54 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:30.000732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101de742-4bda-46f2-b440-f25738514f54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:30.003233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101de742-4bda-46f2-b440-f25738514f54 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:30.023018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 913/1864] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:29 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:30.023018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc3eb3f-9cb0-408f-8f9f-e50ec5ac489b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:30.023018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc3eb3f-9cb0-408f-8f9f-e50ec5ac489b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:30.023018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc3eb3f-9cb0-408f-8f9f-e50ec5ac489b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:30.079567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ddc3eb3f-9cb0-408f-8f9f-e50ec5ac489b None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:30.082506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ddc3eb3f-9cb0-408f-8f9f-e50ec5ac489b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:30.258744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 914/1865] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:52:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:30.259555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc3eb3f-9cb0-408f-8f9f-e50ec5ac489b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:30.273129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc3eb3f-9cb0-408f-8f9f-e50ec5ac489b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:30.279492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:30.280397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:30.280994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:30.281414 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] PATH_INFO: `/v3/projects/02bbc584e55c488f865cdfdcf8b6886f/users/f64c1f5d4b984d4aa5e86d7673f20888/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:30.319560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=02bbc584e55c488f865cdfdcf8b6886f, user_id=f64c1f5d4b984d4aa5e86d7673f20888, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:30.319560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:30.601442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61abe27d9517434bb026799cbff73a34'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:30.601442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61abe27d9517434bb026799cbff73a34'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:30.601442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61abe27d9517434bb026799cbff73a34'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:30.601442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61abe27d9517434bb026799cbff73a34'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:30.823346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61abe27d9517434bb026799cbff73a34'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:30.840783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 915/1866] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:30 2022] PUT /identity/v3/projects/02bbc584e55c488f865cdfdcf8b6886f/users/f64c1f5d4b984d4aa5e86d7673f20888/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 580 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:30.843114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:30.856557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:30.925307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 952/1867] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:29 2022] DELETE /identity/v3/users/61abe27d9517434bb026799cbff73a34 => generated 0 bytes in 1074 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:30.926615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:30.932455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:31.073194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483ca607-9eda-4b40-9501-7dc820e835cc admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:31.103781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564a3b65-6b07-4595-ae48-ede1c68c0cd3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:31.118890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:31.123690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:31.124299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:31.124949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] PATH_INFO: `/v3/projects/cfa63aec061c4a9d8a02ffa80677bf2e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:31.130722 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfa63aec061c4a9d8a02ffa80677bf2e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:31.141365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:31.253143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d02fb47-6ab7-4bf5-8d75-9df6c1959dcf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:31.253143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02fb47-6ab7-4bf5-8d75-9df6c1959dcf service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:31.253143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02fb47-6ab7-4bf5-8d75-9df6c1959dcf service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:31.253143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02fb47-6ab7-4bf5-8d75-9df6c1959dcf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:31.253480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8d02fb47-6ab7-4bf5-8d75-9df6c1959dcf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:31.274007 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d02fb47-6ab7-4bf5-8d75-9df6c1959dcf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:31.275718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d02fb47-6ab7-4bf5-8d75-9df6c1959dcf service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:31.299654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 916/1868] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:31.300967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511eec93-c427-4e32-9b06-b123a36155cb None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:31.301257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511eec93-c427-4e32-9b06-b123a36155cb None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:31.301532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511eec93-c427-4e32-9b06-b123a36155cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:31.375611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-511eec93-c427-4e32-9b06-b123a36155cb None None] MFA Rules not processed for user `f64c1f5d4b984d4aa5e86d7673f20888`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:31.375611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-511eec93-c427-4e32-9b06-b123a36155cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:31.396450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfa63aec061c4a9d8a02ffa80677bf2e', 'request_id': 'req-860fb872-9979-4c09-91b5-827f27bfb917'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:31.405420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfa63aec061c4a9d8a02ffa80677bf2e', 'request_id': 'req-860fb872-9979-4c09-91b5-827f27bfb917'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:31.516864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfa63aec061c4a9d8a02ffa80677bf2e', 'request_id': 'req-860fb872-9979-4c09-91b5-827f27bfb917'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:31.519918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 917/1869] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:31.520780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55405b62-68ca-4a9d-8edb-356bc50b7de4 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:31.521162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55405b62-68ca-4a9d-8edb-356bc50b7de4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:31.528728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55405b62-68ca-4a9d-8edb-356bc50b7de4 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:31.530896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 918/1870] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:52:31 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:52:31.531731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55405b62-68ca-4a9d-8edb-356bc50b7de4 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:31.537327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-55405b62-68ca-4a9d-8edb-356bc50b7de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:31.564178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] The token cache is being invalidate because project cfa63aec061c4a9d8a02ffa80677bf2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:31.564741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:31.568794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:31.574824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 953/1871] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:30 2022] DELETE /identity/v3/projects/cfa63aec061c4a9d8a02ffa80677bf2e => generated 0 bytes in 645 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:31.582877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:31.582877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:31.696745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55405b62-68ca-4a9d-8edb-356bc50b7de4 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:31.702277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4b1c05-8be0-4244-8446-601c08543a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:31.702917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4b1c05-8be0-4244-8446-601c08543a63 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:31.703364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4b1c05-8be0-4244-8446-601c08543a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:31.703636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4b1c05-8be0-4244-8446-601c08543a63 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:31.704285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4b1c05-8be0-4244-8446-601c08543a63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:31.718693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4b1c05-8be0-4244-8446-601c08543a63 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:31.757968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860fb872-9979-4c09-91b5-827f27bfb917 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:31.766944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:31.768682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:31.772194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:31.773425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] PATH_INFO: `/v3/users/6b749dbe9c434d0694d4e18fe35ed800` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:31.779513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b749dbe9c434d0694d4e18fe35ed800)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:31.784006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:31.847819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 919/1872] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:31 2022] POST /identity/v3/projects => generated 379 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:31.857018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:31.862579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:31.863297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:31.979666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:31.984474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:31.984474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:32.015196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:32.033022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b749dbe9c434d0694d4e18fe35ed800'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:32.047373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b749dbe9c434d0694d4e18fe35ed800'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:32.050912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b749dbe9c434d0694d4e18fe35ed800'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:32.057215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b749dbe9c434d0694d4e18fe35ed800'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:32.141653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b749dbe9c434d0694d4e18fe35ed800'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:32.161785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 920/1873] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:52:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:32.164395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:32.166487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:32.182897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 954/1874] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:31 2022] DELETE /identity/v3/users/6b749dbe9c434d0694d4e18fe35ed800 => generated 0 bytes in 607 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:32.235823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:32.237541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:32.355161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79844c65-ab56-47aa-a9b4-338b6e8b5399 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:32.385036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2370240e-9d19-4c82-9568-fe07d6ea078d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:32.401019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:32.401776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:32.402154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:32.402326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:32.432145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:32.434969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:32.454745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 955/1875] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:32.455862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:32.463870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:32.612744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85de924-18c5-4003-9350-d1602359f134 service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:32.625959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:32.626608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:32.626731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:32.626936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:32.627168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:32.627874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:32.628697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:32.629198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:32.629776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] PATH_INFO: `/v3/projects/9583aadcd4274a3e8f9b73d0c3b36910` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:32.629885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:32.635132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9583aadcd4274a3e8f9b73d0c3b36910)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:32.638477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:32.798826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 921/1876] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:32 2022] POST /identity/v3/users => generated 290 bytes in 636 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:32.804541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:32.828609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3896c9-f564-4da4-9359-5dfc26b69b46 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:32.857417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9583aadcd4274a3e8f9b73d0c3b36910', 'request_id': 'req-e957a677-07f1-4080-850e-4cbd568fb775'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:32.867883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9583aadcd4274a3e8f9b73d0c3b36910', 'request_id': 'req-e957a677-07f1-4080-850e-4cbd568fb775'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:32.989819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9583aadcd4274a3e8f9b73d0c3b36910', 'request_id': 'req-e957a677-07f1-4080-850e-4cbd568fb775'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:33.033529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] The token cache is being invalidate because project 9583aadcd4274a3e8f9b73d0c3b36910 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:33.033784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:33.037178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e957a677-07f1-4080-850e-4cbd568fb775 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:33.044561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 956/1877] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:32 2022] DELETE /identity/v3/projects/9583aadcd4274a3e8f9b73d0c3b36910 => generated 0 bytes in 586 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:33.047275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c552a-f311-4201-b42e-0ac8df2c25ec None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.047586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c552a-f311-4201-b42e-0ac8df2c25ec None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.048005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9c552a-f311-4201-b42e-0ac8df2c25ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.069988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:33.070740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.071023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.081059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.082031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:33.085764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:33.085891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ad9c552a-f311-4201-b42e-0ac8df2c25ec None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:33.087522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ad9c552a-f311-4201-b42e-0ac8df2c25ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:33.109570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 957/1878] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:33.111205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a6cede-9942-4b6c-a62b-60c54baa1f7e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.111496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a6cede-9942-4b6c-a62b-60c54baa1f7e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.111773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a6cede-9942-4b6c-a62b-60c54baa1f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.128847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 922/1879] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:32 2022] GET /identity/v3/roles => generated 1420 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:33.143680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a8a6cede-9942-4b6c-a62b-60c54baa1f7e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:33.145126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a8a6cede-9942-4b6c-a62b-60c54baa1f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:33.155654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:33.164322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 958/1880] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:33.167225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:33.169113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a6cede-9942-4b6c-a62b-60c54baa1f7e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:33.217368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a6cede-9942-4b6c-a62b-60c54baa1f7e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:33.229809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:33.233276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83612aab-3bf5-4f99-a57b-d9627fc0395e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:33.237937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.238937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.240251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.241510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:33.246346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:33.249857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:33.263651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:33.266006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.267876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.268156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.268838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:33.277219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:33.287718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 959/1881] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:33.291672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:33.301488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 923/1882] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:33 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:33.302813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:33.312861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6746ba-949f-4dd8-a231-07b7ac54e302 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:33.321992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:33.322386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c814c1-c370-46b4-87a8-8dc093f77b96 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:33.322537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.322701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.322984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.325833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:33.327576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:33.339983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faff17a2-49be-499b-bdd0-ed2210ab0800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:33.340660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faff17a2-49be-499b-bdd0-ed2210ab0800 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.340936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faff17a2-49be-499b-bdd0-ed2210ab0800 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.341218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faff17a2-49be-499b-bdd0-ed2210ab0800 admin admin] PATH_INFO: `/v3/projects/383afa70a41e4056a6275d6058ec5e30/users/209ef52a4a7b4378a3496e370f1e7e48/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.414508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faff17a2-49be-499b-bdd0-ed2210ab0800 admin admin] RBAC: Authorizing `identity:create_grant(project_id=383afa70a41e4056a6275d6058ec5e30, user_id=209ef52a4a7b4378a3496e370f1e7e48, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:33.422058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faff17a2-49be-499b-bdd0-ed2210ab0800 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:33.521110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 960/1883] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:33 2022] POST /identity/v3/projects => generated 381 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:33.524944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:33.550573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 924/1884] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:33 2022] PUT /identity/v3/projects/383afa70a41e4056a6275d6058ec5e30/users/209ef52a4a7b4378a3496e370f1e7e48/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:33.569720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.571301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.571301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.617939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:INSERT=4,keystone:DELETE=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:33.669978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] MFA Rules not processed for user `209ef52a4a7b4378a3496e370f1e7e48`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:33.675557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:33.688311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488897a4-15b8-47b3-b09a-3623a0b87423 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:33.788810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:INSERT=2,keystone:DELETE=3,keystone:SELECT=95 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:33.846111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:33.846853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.847412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.847692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.850233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:33.858125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:33.933255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 925/1885] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:33 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 364 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:33.934189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:33.938712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:33.968416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf0aa5-889a-4974-86c9-e2ddbdb78cb0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:33.983382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:33.984139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:33.984419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:33.985810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] PATH_INFO: `/v3/projects/f6d898b976d64df8977ac7cbc48c4245` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:33.990070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6d898b976d64df8977ac7cbc48c4245)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:33.990355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.048922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 961/1886] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:33 2022] POST /identity/v3/users => generated 291 bytes in 523 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:52:34.055362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.105019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975c1d57-0f93-45a5-8476-28ec11491299 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:34.112205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:34.113231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.113803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.114225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.115938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:34.127542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.170746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6d898b976d64df8977ac7cbc48c4245', 'request_id': 'req-f42244fe-eff4-4845-bae1-ccf6fd090c7e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:34.281176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6d898b976d64df8977ac7cbc48c4245', 'request_id': 'req-f42244fe-eff4-4845-bae1-ccf6fd090c7e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:34.291312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 962/1887] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:34 2022] POST /identity/v3/projects => generated 379 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:34.293105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6d898b976d64df8977ac7cbc48c4245', 'request_id': 'req-f42244fe-eff4-4845-bae1-ccf6fd090c7e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:34.297014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.330386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] The token cache is being invalidate because project f6d898b976d64df8977ac7cbc48c4245 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:34.330898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:34.333447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:34.339233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 926/1888] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:33 2022] DELETE /identity/v3/projects/f6d898b976d64df8977ac7cbc48c4245 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:34.341940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.345553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:34.428655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65365f-0e09-4e6c-8bcc-a230ece2f413 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:34.434150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5437e52e-7ac5-4a2b-851c-6358b1ec982c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:34.435729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437e52e-7ac5-4a2b-851c-6358b1ec982c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.436187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437e52e-7ac5-4a2b-851c-6358b1ec982c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.436606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5437e52e-7ac5-4a2b-851c-6358b1ec982c admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.437500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5437e52e-7ac5-4a2b-851c-6358b1ec982c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:34.439427 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5437e52e-7ac5-4a2b-851c-6358b1ec982c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.454631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 963/1889] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:34 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:34.454631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.454631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.455459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.513887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:34.515604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:34.525753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 964/1890] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:34.526737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.526992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42244fe-eff4-4845-bae1-ccf6fd090c7e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:34.535221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:34.535375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:34.535375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.536004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.536507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.537376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:34.540623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.578723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6ab062-d23f-4c3e-823c-dcd42b3664a6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:34.585266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4722c1c-8d46-44c8-81c9-dd6140ce39f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:34.585266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4722c1c-8d46-44c8-81c9-dd6140ce39f9 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.585266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4722c1c-8d46-44c8-81c9-dd6140ce39f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.586490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4722c1c-8d46-44c8-81c9-dd6140ce39f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.586490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4722c1c-8d46-44c8-81c9-dd6140ce39f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:34.588359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4722c1c-8d46-44c8-81c9-dd6140ce39f9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.612540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 965/1891] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:34 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:34.614106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670820ad-3991-4596-8d21-af221e4764e9 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.614777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670820ad-3991-4596-8d21-af221e4764e9 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.617075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670820ad-3991-4596-8d21-af221e4764e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.669038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 927/1892] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:34 2022] POST /identity/v3/users => generated 296 bytes in 327 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:34.669038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.681658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-670820ad-3991-4596-8d21-af221e4764e9 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:34.683064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-670820ad-3991-4596-8d21-af221e4764e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:34.691538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5f0dd0-5946-46ef-ba13-6f400808ffe3 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:34.709142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 966/1893] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:34.709394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:34.709394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.709394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.710023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670820ad-3991-4596-8d21-af221e4764e9 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.710179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] PATH_INFO: `/v3/projects/07e8870d5fa740b7bb80448df1d53a58/users/0aa61ed820d7435aa0686ed9f72a3d6f/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.729846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670820ad-3991-4596-8d21-af221e4764e9 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:34.742807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:34.744632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.744916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.745188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.745847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:34.746647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07e8870d5fa740b7bb80448df1d53a58, user_id=0aa61ed820d7435aa0686ed9f72a3d6f, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:34.750450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.752253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.772096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 967/1894] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:34 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:34.774086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.792704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd992d87-175a-4b3e-be53-4f583408aa96 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:34.809443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b7c253-ee80-4f37-b3e7-52460316c53b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:34.810099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7c253-ee80-4f37-b3e7-52460316c53b admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.810396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7c253-ee80-4f37-b3e7-52460316c53b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.810694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7c253-ee80-4f37-b3e7-52460316c53b admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.811357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b7c253-ee80-4f37-b3e7-52460316c53b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:34.816635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b7c253-ee80-4f37-b3e7-52460316c53b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:34.818503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b7c253-ee80-4f37-b3e7-52460316c53b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:34.855423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 928/1895] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:34 2022] PUT /identity/v3/projects/07e8870d5fa740b7bb80448df1d53a58/users/0aa61ed820d7435aa0686ed9f72a3d6f/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:34.855423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:34.874781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 968/1896] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:34.877069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc317c-6eba-4d57-bb13-4d9ce5ea3588 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:34.877679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc317c-6eba-4d57-bb13-4d9ce5ea3588 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:34.878171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc317c-6eba-4d57-bb13-4d9ce5ea3588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:34.939149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1adc317c-6eba-4d57-bb13-4d9ce5ea3588 None None] MFA Rules not processed for user `0aa61ed820d7435aa0686ed9f72a3d6f`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:34.940872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1adc317c-6eba-4d57-bb13-4d9ce5ea3588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:35.063089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc7dc8-2d86-41d3-9280-3204bda670d8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.079521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.081966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.082595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.082944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] PATH_INFO: `/v3/projects/75817b1bc85049b3bb2be199afc53e8f/users/a10d1d17097d4233968634079b0a31d0/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.130077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=75817b1bc85049b3bb2be199afc53e8f, user_id=a10d1d17097d4233968634079b0a31d0, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.134462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.138008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 969/1897] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:34 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:35.138008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adc317c-6eba-4d57-bb13-4d9ce5ea3588 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:35.171284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adc317c-6eba-4d57-bb13-4d9ce5ea3588 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.229822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 929/1898] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:34 2022] PUT /identity/v3/projects/75817b1bc85049b3bb2be199afc53e8f/users/a10d1d17097d4233968634079b0a31d0/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:35.230799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.231661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:35.232927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.233586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.233901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.237146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.249465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.255165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:35.342212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:35.398189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f612619-dba5-41fd-962f-5cf6a2cfdf8d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.403865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.404553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.404833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.405108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.405865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.407876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.413894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 970/1899] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:35 2022] POST /identity/v3/projects => generated 409 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:35.415048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:35.576051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e5f1df-65da-4039-8d6d-c51719358100 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.584004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 930/1900] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:35 2022] POST /identity/v3/projects => generated 381 bytes in 353 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:35.588881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:35.589663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.591872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.591872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.591872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.592936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.599689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.619987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 971/1901] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:35 2022] GET /identity/v3/roles => generated 1420 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:35.623944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:35.640001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86158b43-2a4a-408a-8af1-5812e7499076 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.719690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2192f323-0741-43c8-9062-d18ef7d95e86 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.725214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.726163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.726780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.727048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.727970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.731232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.751089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.751714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.752583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.752583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.752989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.773263 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.865473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 931/1902] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:35 2022] POST /identity/v3/users => generated 305 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:35.867322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:35.869811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:35.891358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 972/1903] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:35 2022] POST /identity/v3/users => generated 297 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:35.895633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:35.895759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144033e0-7d3c-42d3-ac33-e2780fe508c3 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.902119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.903522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.903894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.904235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] PATH_INFO: `/v3/users/2878f48cff82499ab41c2237b3afdaa2` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.905721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2878f48cff82499ab41c2237b3afdaa2)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.908464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.930136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a47705-8bae-4a03-92ce-464798793c23 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:35.941928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:35.946065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:35.946611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:35.946893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] PATH_INFO: `/v3/projects/75817b1bc85049b3bb2be199afc53e8f/users/a10d1d17097d4233968634079b0a31d0/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:35.955222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75817b1bc85049b3bb2be199afc53e8f, user_id=a10d1d17097d4233968634079b0a31d0, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:35.962725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:35.985462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 973/1904] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:35 2022] PUT /identity/v3/projects/75817b1bc85049b3bb2be199afc53e8f/users/a10d1d17097d4233968634079b0a31d0/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:35.993138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:36.103064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0379d7-4806-4314-8def-c8b33e490180 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:36.109779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:36.110430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.110763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.111052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:36.111769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:36.113585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:36.130622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 974/1905] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:35 2022] GET /identity/v3/roles => generated 1420 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:36.136389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:36.149685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d431c2-5916-491e-9c2b-fd3210e3d112 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:36.175010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dbd8cd-f02f-4acb-b849-64ed2142b812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:36.175010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dbd8cd-f02f-4acb-b849-64ed2142b812 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.175010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dbd8cd-f02f-4acb-b849-64ed2142b812 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.175010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dbd8cd-f02f-4acb-b849-64ed2142b812 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:36.175395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dbd8cd-f02f-4acb-b849-64ed2142b812 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:36.175395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dbd8cd-f02f-4acb-b849-64ed2142b812 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:36.181149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2878f48cff82499ab41c2237b3afdaa2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:36.185264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 975/1906] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:36 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:36.188995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.188995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.188995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:36.198111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2878f48cff82499ab41c2237b3afdaa2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:36.199112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2878f48cff82499ab41c2237b3afdaa2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:36.200939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2878f48cff82499ab41c2237b3afdaa2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:36.245220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] MFA Rules not processed for user `a10d1d17097d4233968634079b0a31d0`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:36.245220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:36.293367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2878f48cff82499ab41c2237b3afdaa2'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:36.334585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 932/1907] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:35 2022] DELETE /identity/v3/users/2878f48cff82499ab41c2237b3afdaa2 => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:36.334585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:36.338938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:36.437418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:36.439293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 976/1908] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:36 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:36.440073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:36.455828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:36.490448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393417e4-c42f-43da-ae9c-13832f9ffc47 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:36.548868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083aece8-6814-4135-a647-1bd5114219e2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:36.589029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c22435f-28f5-4c3a-9f9f-3902502bfcf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:36.591093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22435f-28f5-4c3a-9f9f-3902502bfcf3 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.591408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22435f-28f5-4c3a-9f9f-3902502bfcf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.591653 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22435f-28f5-4c3a-9f9f-3902502bfcf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:36.592326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c22435f-28f5-4c3a-9f9f-3902502bfcf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:36.593940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c22435f-28f5-4c3a-9f9f-3902502bfcf3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:36.621194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 933/1909] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:36 2022] GET /identity/v3/roles => generated 1420 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:36.622474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.622768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.623094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:36.627542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:36.651616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:36.652693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.653075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.653425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] PATH_INFO: `/v3/projects/71cc499c88e94101b52d535d4189b6ad/users/e80b199f8ed848beaf64ebaf1d745a3d/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:36.653695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:36.655137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:36.670329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 934/1910] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:36.670329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:36.671333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:36.680393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] RBAC: Authorizing `identity:create_grant(project_id=71cc499c88e94101b52d535d4189b6ad, user_id=e80b199f8ed848beaf64ebaf1d745a3d, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:36.686210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:36.785444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 977/1911] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:36 2022] PUT /identity/v3/projects/71cc499c88e94101b52d535d4189b6ad/users/e80b199f8ed848beaf64ebaf1d745a3d/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:36.787123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:36.806980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d0c63c-d6c6-44d9-b360-c84a5bd7a57b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:36.905774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:36.908842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.910469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.910755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:36.942291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e820af-ef3b-4dca-9bbe-f2ce9ad0165b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:36.962591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:36.963288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:36.963566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:36.963840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] PATH_INFO: `/v3/projects/6be9611496034f7d981daac6762ac33f/users/17a61a5c1ec348d196890c40fc11bf3b/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.002132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6be9611496034f7d981daac6762ac33f, user_id=17a61a5c1ec348d196890c40fc11bf3b, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:37.004851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:37.099161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 978/1912] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:36 2022] PUT /identity/v3/projects/6be9611496034f7d981daac6762ac33f/users/17a61a5c1ec348d196890c40fc11bf3b/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:37.099161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:37.099161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:37.129817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:37.133035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:37.161343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 935/1913] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:36 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:37.162415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:37.184561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9e633-119e-4691-8087-91ae80f7621f service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:37.266042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e60022-353e-4752-9d73-c5b6732ddda5 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:37.273453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:37.274099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:37.274395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:37.274678 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.305732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1399a9b-c8d9-41f3-9057-8c7b0b3556a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:37.317857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1399a9b-c8d9-41f3-9057-8c7b0b3556a9 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:37.318285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1399a9b-c8d9-41f3-9057-8c7b0b3556a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:37.318592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1399a9b-c8d9-41f3-9057-8c7b0b3556a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.319995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1399a9b-c8d9-41f3-9057-8c7b0b3556a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:37.324059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1399a9b-c8d9-41f3-9057-8c7b0b3556a9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:37.383527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 936/1914] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:37 2022] GET /identity/v3/roles => generated 1420 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:37.401209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6cb0b6-84bb-4424-b80b-de4639d196dc None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:37.401615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6cb0b6-84bb-4424-b80b-de4639d196dc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:37.402174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6cb0b6-84bb-4424-b80b-de4639d196dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.434428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:37.458396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:37.468108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:37.472054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8e6cb0b6-84bb-4424-b80b-de4639d196dc None None] MFA Rules not processed for user `17a61a5c1ec348d196890c40fc11bf3b`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:37.476250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8e6cb0b6-84bb-4424-b80b-de4639d196dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:37.482854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 979/1915] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:37.484269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:37.485788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:37.526437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:37.620887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 937/1916] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:37 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:37.622128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6cb0b6-84bb-4424-b80b-de4639d196dc None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:37.625472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208255bd-cb52-4536-9d79-21e1a28b1947 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:37.638076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f3fc2e-a89c-467f-af59-9d024d0a845c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:37.638779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3fc2e-a89c-467f-af59-9d024d0a845c service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:37.639061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3fc2e-a89c-467f-af59-9d024d0a845c service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:37.639330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f3fc2e-a89c-467f-af59-9d024d0a845c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.651805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6cb0b6-84bb-4424-b80b-de4639d196dc None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:37.653350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-46f3fc2e-a89c-467f-af59-9d024d0a845c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:37.660004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:37.660004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:37.660004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:37.660004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] PATH_INFO: `/v3/projects/71cc499c88e94101b52d535d4189b6ad/users/e80b199f8ed848beaf64ebaf1d745a3d/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.661151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71cc499c88e94101b52d535d4189b6ad, user_id=e80b199f8ed848beaf64ebaf1d745a3d, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:37.668115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:37.692645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 938/1917] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:37 2022] PUT /identity/v3/projects/71cc499c88e94101b52d535d4189b6ad/users/e80b199f8ed848beaf64ebaf1d745a3d/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:37.693881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:37.757403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f3fc2e-a89c-467f-af59-9d024d0a845c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:37.759007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f3fc2e-a89c-467f-af59-9d024d0a845c service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:37.769811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 980/1918] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:37 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:37.773850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ed6b4-4495-4388-9eee-b30fadccb89c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:37.774247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ed6b4-4495-4388-9eee-b30fadccb89c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:37.774648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ed6b4-4495-4388-9eee-b30fadccb89c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.837912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-418ed6b4-4495-4388-9eee-b30fadccb89c None None] MFA Rules not processed for user `e80b199f8ed848beaf64ebaf1d745a3d`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:37.839634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-418ed6b4-4495-4388-9eee-b30fadccb89c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:37.856409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6453d7e3-a461-49ba-9608-6bc553eeb1c1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:37.862853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:37.875461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:37.876184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:37.876553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:37.880108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:37.880108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:37.880108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:37.950239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 981/1919] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:37 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:37.962818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ed6b4-4495-4388-9eee-b30fadccb89c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.008638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 939/1920] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:37 2022] POST /identity/v3/projects => generated 405 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:38.012985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.030282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ed6b4-4495-4388-9eee-b30fadccb89c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:38.085641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:38.090629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:38.091273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.091693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.091972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.098379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37916acf-b984-497c-80b1-1354842ffcbe admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:38.104231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45459468-651d-4743-bd74-489415e231cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:38.105181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45459468-651d-4743-bd74-489415e231cd admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.105562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45459468-651d-4743-bd74-489415e231cd admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.105935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45459468-651d-4743-bd74-489415e231cd admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.107089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45459468-651d-4743-bd74-489415e231cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:38.109289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45459468-651d-4743-bd74-489415e231cd admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:38.188143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:38.190760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:38.208206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 982/1921] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:37 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:38.235267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 940/1922] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:38 2022] POST /identity/v3/users => generated 309 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:38.236663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.258141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec47d37-4d52-4103-aaaa-d8c2dd796d3d service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:38.278103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:38.278776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.279105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.279387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.280052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:38.289112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:38.307841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 983/1923] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:38 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:38.310093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45459468-651d-4743-bd74-489415e231cd admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.336050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45459468-651d-4743-bd74-489415e231cd admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:38.355270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:38.355270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.355270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.355270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] PATH_INFO: `/v3/projects/a43da02339fb4b238fccc578924ab426/users/6d02b613b554431bac7429a0edef88d5/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.380035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:38.399463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a43da02339fb4b238fccc578924ab426, user_id=6d02b613b554431bac7429a0edef88d5, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:38.404390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:38.545817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 941/1924] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:38 2022] PUT /identity/v3/projects/a43da02339fb4b238fccc578924ab426/users/6d02b613b554431bac7429a0edef88d5/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:38.548644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.565582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529fccaf-fbca-4d03-bd18-555bb3654872 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:38.607276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.613801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:38.656025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:38.657973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.658267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.658574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.659322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:38.662541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:38.680934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 984/1925] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:38 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:38.695993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.723143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32733297-d254-4bf8-9c7b-3dc61aff9b1b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:38.723460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39194034-19e8-43e9-802c-d5502a6132a6 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:38.723460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3dc9a6-441a-4e12-a40b-a076fbda8284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:38.723460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3dc9a6-441a-4e12-a40b-a076fbda8284 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.723460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3dc9a6-441a-4e12-a40b-a076fbda8284 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.723756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3dc9a6-441a-4e12-a40b-a076fbda8284 admin admin] PATH_INFO: `/v3/projects/a43da02339fb4b238fccc578924ab426/users/6d02b613b554431bac7429a0edef88d5/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.724143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:38.725033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.725430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.725712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3dc9a6-441a-4e12-a40b-a076fbda8284 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a43da02339fb4b238fccc578924ab426, user_id=6d02b613b554431bac7429a0edef88d5, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:38.725882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] PATH_INFO: `/v3/projects/9733341358a3481388a52c4724ed053d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.728145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9733341358a3481388a52c4724ed053d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:38.728757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3dc9a6-441a-4e12-a40b-a076fbda8284 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:38.731058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:38.751450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 985/1926] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:38 2022] PUT /identity/v3/projects/a43da02339fb4b238fccc578924ab426/users/6d02b613b554431bac7429a0edef88d5/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:38.758611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bdbd0c-a0b9-45e6-987a-73f93b57277f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:38.758898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bdbd0c-a0b9-45e6-987a-73f93b57277f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:38.759191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bdbd0c-a0b9-45e6-987a-73f93b57277f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:38.798298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-09bdbd0c-a0b9-45e6-987a-73f93b57277f None None] MFA Rules not processed for user `6d02b613b554431bac7429a0edef88d5`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:38.799952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-09bdbd0c-a0b9-45e6-987a-73f93b57277f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:38.889868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 986/1927] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:38 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:38.906960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9733341358a3481388a52c4724ed053d', 'request_id': 'req-40960477-2082-420f-8433-08f5b93ebf22'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:38.921385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:38.927297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bdbd0c-a0b9-45e6-987a-73f93b57277f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:38.983336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9733341358a3481388a52c4724ed053d', 'request_id': 'req-40960477-2082-420f-8433-08f5b93ebf22'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:38.984999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9733341358a3481388a52c4724ed053d', 'request_id': 'req-40960477-2082-420f-8433-08f5b93ebf22'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:38.998596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] The token cache is being invalidate because project 9733341358a3481388a52c4724ed053d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:38.998890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:38.999844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.004551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 942/1928] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:38 2022] DELETE /identity/v3/projects/9733341358a3481388a52c4724ed053d => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:39.004551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:39.004551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:39.015404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bdbd0c-a0b9-45e6-987a-73f93b57277f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:39.070721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:39.071346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:39.071624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:39.071991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:39.073646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:39.091497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40960477-2082-420f-8433-08f5b93ebf22 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:39.101161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:39.108254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:39.108254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:39.108254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] PATH_INFO: `/v3/users/0aa61ed820d7435aa0686ed9f72a3d6f` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:39.108254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0aa61ed820d7435aa0686ed9f72a3d6f)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:39.108254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:39.144422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:39.145859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:39.156731 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 987/1929] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:38 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:39.157631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:39.159001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:39.174507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49d8dd-e9d4-4faf-ac1e-2bba788dc594 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:39.179497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:39.180497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:39.180777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:39.181114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] PATH_INFO: `/v3/users/52b3a51f2e2b474385cfdb519dea679c` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:39.183328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] RBAC: Authorizing `identity:delete_user(user_id=52b3a51f2e2b474385cfdb519dea679c)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:39.185377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:39.298760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0aa61ed820d7435aa0686ed9f72a3d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.308200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0aa61ed820d7435aa0686ed9f72a3d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.308906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0aa61ed820d7435aa0686ed9f72a3d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.309529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0aa61ed820d7435aa0686ed9f72a3d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.376799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52b3a51f2e2b474385cfdb519dea679c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.384307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52b3a51f2e2b474385cfdb519dea679c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.385003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52b3a51f2e2b474385cfdb519dea679c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.385619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52b3a51f2e2b474385cfdb519dea679c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.435640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c9e53eb1-9bc1-45a2-b020-0eb657bcf629 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0aa61ed820d7435aa0686ed9f72a3d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.453809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 943/1930] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:39 2022] DELETE /identity/v3/users/0aa61ed820d7435aa0686ed9f72a3d6f => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:39.458033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:39.458319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:39.458620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:39.477746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:39.478986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:39.537201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 944/1931] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:39.547154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:39.548754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:39.550494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52b3a51f2e2b474385cfdb519dea679c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:39.590929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 988/1932] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:39 2022] DELETE /identity/v3/users/52b3a51f2e2b474385cfdb519dea679c => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:39.650060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a1f25c-e10d-4ecc-ba2f-ccff98fdc600 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:39.704508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:39.706240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:39.711641 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:39.712440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:39.712706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:39.713045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:39.774698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4176e471-aa49-4fa1-b228-6f6baedf714d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:39.778007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:39.778762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:39.779213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:39.779491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:39.781896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:39.786182 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:39.788207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:39.799740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 945/1933] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:39.853699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:39.856867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:39.867137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 989/1934] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:39 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:39.927015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=90,keystone:DELETE=4,keystone:INSERT=6 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:40.376815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:40.376815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:40.403332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdeb4f1-8797-48d2-bf66-48449b99e07d service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:40.408731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:40.413394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:40.413394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:40.413394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] PATH_INFO: `/v3/projects/07e8870d5fa740b7bb80448df1d53a58` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:40.413394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07e8870d5fa740b7bb80448df1d53a58)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:40.414310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:40.567494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:40.569797 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:40.587713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=1,keystone:SELECT=94,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:40.592307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07e8870d5fa740b7bb80448df1d53a58', 'request_id': 'req-1f487b28-e283-4241-b086-ec8586e4fc35'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.597858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1237282-5788-442f-a17f-78071fa9af3a service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:40.667894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:40.668524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:40.668799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:40.669068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] PATH_INFO: `/v3/projects/98b8b827e137460b8f138df67bfb589a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:40.671123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07e8870d5fa740b7bb80448df1d53a58', 'request_id': 'req-1f487b28-e283-4241-b086-ec8586e4fc35'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.682175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] RBAC: Authorizing `identity:delete_project(project_id=98b8b827e137460b8f138df67bfb589a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:40.682175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:40.682494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07e8870d5fa740b7bb80448df1d53a58', 'request_id': 'req-1f487b28-e283-4241-b086-ec8586e4fc35'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.691805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] The token cache is being invalidate because project 07e8870d5fa740b7bb80448df1d53a58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:40.692098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.692745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.695211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 946/1935] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:40 2022] DELETE /identity/v3/projects/07e8870d5fa740b7bb80448df1d53a58 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:40.699043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:40.700394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:40.716088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f487b28-e283-4241-b086-ec8586e4fc35 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:40.722031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:40.722706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:40.722995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:40.723267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] PATH_INFO: `/v3/users/e80b199f8ed848beaf64ebaf1d745a3d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:40.724403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e80b199f8ed848beaf64ebaf1d745a3d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:40.726530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:40.808739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98b8b827e137460b8f138df67bfb589a', 'request_id': 'req-35596bd4-64a9-463c-b718-d4226f99486a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.815186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98b8b827e137460b8f138df67bfb589a', 'request_id': 'req-35596bd4-64a9-463c-b718-d4226f99486a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.954562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98b8b827e137460b8f138df67bfb589a', 'request_id': 'req-35596bd4-64a9-463c-b718-d4226f99486a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.972574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] The token cache is being invalidate because project 98b8b827e137460b8f138df67bfb589a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:40.972912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.973547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-35596bd4-64a9-463c-b718-d4226f99486a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:40.976255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 990/1936] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:40 2022] DELETE /identity/v3/projects/98b8b827e137460b8f138df67bfb589a => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:40.980834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bee026-6745-4d0e-8b29-f09d7deb5da0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:40.981183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bee026-6745-4d0e-8b29-f09d7deb5da0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:40.981576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bee026-6745-4d0e-8b29-f09d7deb5da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.010576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c4bee026-6745-4d0e-8b29-f09d7deb5da0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:41.011510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c4bee026-6745-4d0e-8b29-f09d7deb5da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:41.025298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e80b199f8ed848beaf64ebaf1d745a3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:41.035144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e80b199f8ed848beaf64ebaf1d745a3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:41.035985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e80b199f8ed848beaf64ebaf1d745a3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:41.036811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e80b199f8ed848beaf64ebaf1d745a3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:41.106424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 991/1937] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:41.114216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed1558-af49-4f03-b49b-1dd08741fa52 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:41.114704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed1558-af49-4f03-b49b-1dd08741fa52 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:41.116493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed1558-af49-4f03-b49b-1dd08741fa52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.117067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e80b199f8ed848beaf64ebaf1d745a3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:41.139886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 947/1938] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:40 2022] DELETE /identity/v3/users/e80b199f8ed848beaf64ebaf1d745a3d => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:41.141643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-15ed1558-af49-4f03-b49b-1dd08741fa52 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:41.142902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-15ed1558-af49-4f03-b49b-1dd08741fa52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:41.278870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 992/1939] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:41.283398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:41.295688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763c2a28-5c1f-488f-ae26-2a788ed22c0c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:41.300786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:41.301438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:41.301714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:41.301981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.302664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:41.302963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:41.305003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:41.319921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 948/1940] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:41.323368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed1558-af49-4f03-b49b-1dd08741fa52 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:41.333200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed1558-af49-4f03-b49b-1dd08741fa52 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:41.339330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:41.339976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:41.340262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:41.340530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.341176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:41.343172 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:41.699727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 993/1941] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:41 2022] POST /identity/v3/projects => generated 373 bytes in 337 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:41.702434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:41.702434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb272666-43b5-48fc-8fca-3828e494b497 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:41.704552 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:41.712408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:41.714590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:41.763167 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:41.763852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:41.764108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:41.764378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.765024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:41.767364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:41.805671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477754b6-96f5-4e11-a6c7-e0e339aedc9b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:41.813527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:41.814233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:41.814547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:41.814824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] PATH_INFO: `/v3/projects/71cc499c88e94101b52d535d4189b6ad` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.817379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] RBAC: Authorizing `identity:delete_project(project_id=71cc499c88e94101b52d535d4189b6ad)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:41.820208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:41.838852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 949/1942] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:41 2022] POST /identity/v3/users => generated 287 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:41.844678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:41.866941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bbaa5c-31db-4387-852c-f3e4a0ae0aa5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:41.875357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:41.877191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:41.877520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:41.877799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.878555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:41.880887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:41.900281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 950/1943] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:41 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:41.908582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:41.923976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add2a593-9dd5-4714-9a5e-6c2eb0b83235 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:41.931080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:41.934262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:41.934262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:41.934262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:41.934262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:41.934916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:41.946887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 951/1944] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:41 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:41.953664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:41.966035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac2cdf1-9fcf-4352-8061-835f67030108 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:41.977958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71cc499c88e94101b52d535d4189b6ad', 'request_id': 'req-349d07ee-83cf-4a21-9664-0053b0d5354e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:41.990236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71cc499c88e94101b52d535d4189b6ad', 'request_id': 'req-349d07ee-83cf-4a21-9664-0053b0d5354e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:42.034060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9c52f4-5b94-4032-a806-161a6337ba25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.034761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9c52f4-5b94-4032-a806-161a6337ba25 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.035082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9c52f4-5b94-4032-a806-161a6337ba25 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.035359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9c52f4-5b94-4032-a806-161a6337ba25 admin admin] PATH_INFO: `/v3/projects/894b70b0c4954b869c836a087285825a/users/b0c2529a3b9e41e084c87c6be9ae22ab/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.050587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9c52f4-5b94-4032-a806-161a6337ba25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=894b70b0c4954b869c836a087285825a, user_id=b0c2529a3b9e41e084c87c6be9ae22ab, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.053347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9c52f4-5b94-4032-a806-161a6337ba25 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.082836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71cc499c88e94101b52d535d4189b6ad', 'request_id': 'req-349d07ee-83cf-4a21-9664-0053b0d5354e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:42.095790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] The token cache is being invalidate because project 71cc499c88e94101b52d535d4189b6ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:42.096135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:42.096796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-349d07ee-83cf-4a21-9664-0053b0d5354e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:42.098522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 994/1945] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:41 2022] DELETE /identity/v3/projects/71cc499c88e94101b52d535d4189b6ad => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:42.104350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952c4e66-1085-4037-a213-d16c98069462 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.104640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952c4e66-1085-4037-a213-d16c98069462 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.104942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952c4e66-1085-4037-a213-d16c98069462 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.126354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-952c4e66-1085-4037-a213-d16c98069462 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:42.129868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-952c4e66-1085-4037-a213-d16c98069462 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:42.139359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 952/1946] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:41 2022] PUT /identity/v3/projects/894b70b0c4954b869c836a087285825a/users/b0c2529a3b9e41e084c87c6be9ae22ab/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:42.145463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.145809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.146279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.176243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:42.177766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:42.222567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 995/1947] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:42.227599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eb3fe8-3346-45d5-9414-eaca34eb21d3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.227905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eb3fe8-3346-45d5-9414-eaca34eb21d3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.228747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eb3fe8-3346-45d5-9414-eaca34eb21d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.249066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-81eb3fe8-3346-45d5-9414-eaca34eb21d3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:42.254474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 953/1948] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:42.259238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-81eb3fe8-3346-45d5-9414-eaca34eb21d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:42.268206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.270966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:42.276754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 996/1949] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:42.277872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eb3fe8-3346-45d5-9414-eaca34eb21d3 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.291258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eb3fe8-3346-45d5-9414-eaca34eb21d3 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.300284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.300284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.300284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.300284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.300696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:42.300696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.301445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.319195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 997/1950] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:42.320374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.331152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ec8fc-61de-4fe9-9205-7f2960af2ffa admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.337563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.338177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.338490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.338768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.339705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.341687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.383881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897df63-aad8-4cf1-9ea8-124f140f34da None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.388309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.391113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.391113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.391113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.405347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.407080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.418396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 954/1951] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:42.443009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 998/1952] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:42 2022] POST /identity/v3/projects => generated 383 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:42.444097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.454307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edad7472-6a6b-4664-8a28-baa5a81c4865 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.534784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.535479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.536049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.536993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.537238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.540384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.557587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:42.643594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 955/1953] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:42 2022] POST /identity/v3/users => generated 292 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:42.644520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.658247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d1f913-dd2c-4e40-9f4d-f1843b0a87b1 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.665127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.666163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.666715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.667488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.668462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.670532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.684535 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 999/1954] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:42 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:42.688846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.719559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4ad4bc-9b5f-41de-9449-b83830067b30 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.734272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.735093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.735918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.736300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.737127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.742291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.766645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 956/1955] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:42 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:42.768040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.780298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f762d1-56eb-4577-a5e8-76acdf67d7cf admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.785748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254874b8-02fc-41db-b9fc-a8e1cf75391a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.786492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254874b8-02fc-41db-b9fc-a8e1cf75391a admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.786783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254874b8-02fc-41db-b9fc-a8e1cf75391a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.787043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254874b8-02fc-41db-b9fc-a8e1cf75391a admin admin] PATH_INFO: `/v3/projects/35822fc8958f43d997f535fcb6a5aea6/users/0ee689c80ad34b189e6d87c4cb7fb1c7/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.804294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254874b8-02fc-41db-b9fc-a8e1cf75391a admin admin] RBAC: Authorizing `identity:create_grant(project_id=35822fc8958f43d997f535fcb6a5aea6, user_id=0ee689c80ad34b189e6d87c4cb7fb1c7, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.807409 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254874b8-02fc-41db-b9fc-a8e1cf75391a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.849002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:42.851120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:42.865510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc4b603-605e-4752-abb0-4693cc936a9a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:42.871332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:42.871975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.872250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.872520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] PATH_INFO: `/v3/users/f64c1f5d4b984d4aa5e86d7673f20888` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.873716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f64c1f5d4b984d4aa5e86d7673f20888)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:42.875816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:42.922497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1000/1956] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:42 2022] PUT /identity/v3/projects/35822fc8958f43d997f535fcb6a5aea6/users/0ee689c80ad34b189e6d87c4cb7fb1c7/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:42.933723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f886bb-a0eb-4322-91fe-41d4dd1afd63 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:42.934579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f886bb-a0eb-4322-91fe-41d4dd1afd63 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:42.935092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f886bb-a0eb-4322-91fe-41d4dd1afd63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:42.949583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:42.978157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b2f886bb-a0eb-4322-91fe-41d4dd1afd63 None None] MFA Rules not processed for user `0ee689c80ad34b189e6d87c4cb7fb1c7`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:42.979783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b2f886bb-a0eb-4322-91fe-41d4dd1afd63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:43.051057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1001/1957] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:43.056333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f886bb-a0eb-4322-91fe-41d4dd1afd63 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.119960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f64c1f5d4b984d4aa5e86d7673f20888'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:43.123203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f886bb-a0eb-4322-91fe-41d4dd1afd63 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.126901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f64c1f5d4b984d4aa5e86d7673f20888'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:43.127591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f64c1f5d4b984d4aa5e86d7673f20888'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:43.127905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.128293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f64c1f5d4b984d4aa5e86d7673f20888'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:43.128504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.128790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.129064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.129693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:43.132039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:43.214647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8d771428-253c-4271-b455-c7ec2b57e893 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f64c1f5d4b984d4aa5e86d7673f20888'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:43.235843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 957/1958] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:42 2022] DELETE /identity/v3/users/f64c1f5d4b984d4aa5e86d7673f20888 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:43.237630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.237915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.238187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.264127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:43.264455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:43.267323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1002/1959] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:43 2022] POST /identity/v3/projects => generated 383 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:43.271232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.273941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:43.339090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 958/1960] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:43.343625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af87ba-0921-490c-8b6a-518c7fce439e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.348368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.348982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.349258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.354259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.354259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:43.354259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:43.354547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.355125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:43.416250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4575f6b-e56d-45d6-a083-02cad7a3c277 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.419865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.420473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.420757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.421011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.431223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:43.432723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:43.442557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 959/1961] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:43.470854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1003/1962] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:43 2022] POST /identity/v3/users => generated 298 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:43.474805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.485032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d605c3b-7662-4321-86d7-b9650c2c5967 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.489694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.490227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.490537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.491400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.492345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:43.493984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:43.504038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 960/1963] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:43.508754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.519167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81846dc6-0ea3-4b92-ac22-4f20860e375a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.524589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.525254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.525534 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.525806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] PATH_INFO: `/v3/projects/ea594acdd47243128163b58b12761733/users/02f30723ae9e4ad99e2a751d3fc8704f/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.542472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea594acdd47243128163b58b12761733, user_id=02f30723ae9e4ad99e2a751d3fc8704f, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:43.546806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:43.622973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1004/1964] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:43 2022] PUT /identity/v3/projects/ea594acdd47243128163b58b12761733/users/02f30723ae9e4ad99e2a751d3fc8704f/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:43.627105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.638470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2e5a94-dc97-4ff0-b4e4-7f664a156ee8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.693936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4060eca-160d-4248-b8b4-657e4d1da480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.694752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4060eca-160d-4248-b8b4-657e4d1da480 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.695049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4060eca-160d-4248-b8b4-657e4d1da480 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.695299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4060eca-160d-4248-b8b4-657e4d1da480 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.695966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4060eca-160d-4248-b8b4-657e4d1da480 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:43.697579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4060eca-160d-4248-b8b4-657e4d1da480 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:43.707760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 961/1965] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:43.713618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.728155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a892aca2-f4e7-43da-8627-51cb37cbf39d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.734623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.734623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.734623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.734623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] PATH_INFO: `/v3/projects/ea594acdd47243128163b58b12761733/users/02f30723ae9e4ad99e2a751d3fc8704f/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.736864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea594acdd47243128163b58b12761733, user_id=02f30723ae9e4ad99e2a751d3fc8704f, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:43.739827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:43.753253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1005/1966] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:43 2022] PUT /identity/v3/projects/ea594acdd47243128163b58b12761733/users/02f30723ae9e4ad99e2a751d3fc8704f/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:43.765206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6503f7e9-600a-4ba7-bd4c-d36017c7010b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.765808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6503f7e9-600a-4ba7-bd4c-d36017c7010b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.765808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6503f7e9-600a-4ba7-bd4c-d36017c7010b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:43.818930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6503f7e9-600a-4ba7-bd4c-d36017c7010b None None] MFA Rules not processed for user `02f30723ae9e4ad99e2a751d3fc8704f`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:43.821850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6503f7e9-600a-4ba7-bd4c-d36017c7010b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:43.900685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 962/1967] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:43 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:43.916821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:43.918879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:43.923871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:43.980677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe78fc1-90df-47ff-9ac4-59bf79d666e6 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:43.994105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:43.994872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:43.995156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:43.995430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.016024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.016024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:44.029417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1006/1968] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:43 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:44.079254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.079416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.079650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.126787 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:44.133480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] MFA Rules not processed for user `b0c2529a3b9e41e084c87c6be9ae22ab`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:44.134860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:44.212868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:44.214111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 963/1969] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:44.220006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:44.220701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:44.223248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:44.333080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7f9363-360e-408b-b8d1-d323e02480c6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:44.339029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:44.340279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.340279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.340279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.341182 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.345825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:44.404496 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97f5879-7546-45ab-bc5e-2007cf0450ec service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:44.409365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:44.410980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.410980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.410980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] PATH_INFO: `/v3/projects/02bbc584e55c488f865cdfdcf8b6886f` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.411837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02bbc584e55c488f865cdfdcf8b6886f)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.414762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:44.460859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:44.498631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 964/1970] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:44 2022] POST /identity/v3/projects => generated 373 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:44.498631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:44.545860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:44.555850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d78b601-108d-4c32-8550-ee2eef9d26fb admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:44.565635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02bbc584e55c488f865cdfdcf8b6886f', 'request_id': 'req-b9b5e412-4eab-4d27-aa01-37e8d32126d9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:44.573980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02bbc584e55c488f865cdfdcf8b6886f', 'request_id': 'req-b9b5e412-4eab-4d27-aa01-37e8d32126d9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:44.613244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:44.613869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.614143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.614461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.616534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:44.649207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02bbc584e55c488f865cdfdcf8b6886f', 'request_id': 'req-b9b5e412-4eab-4d27-aa01-37e8d32126d9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:44.664006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] The token cache is being invalidate because project 02bbc584e55c488f865cdfdcf8b6886f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:44.664611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:44.665344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:44.667089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1007/1971] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:44 2022] DELETE /identity/v3/projects/02bbc584e55c488f865cdfdcf8b6886f => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:44.670841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:44.672295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:44.685478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.687131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:44.697819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 965/1972] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:44.698178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:44.755175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5e412-4eab-4d27-aa01-37e8d32126d9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:44.755175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:44.764088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:44.765114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.765558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.765838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.766513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.768712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:44.779794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31a2db0-5c32-4ec7-b174-497d6457fe61 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:44.784605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:44.785236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.785514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.785786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] PATH_INFO: `/v3/users/209ef52a4a7b4378a3496e370f1e7e48` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.786942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] RBAC: Authorizing `identity:delete_user(user_id=209ef52a4a7b4378a3496e370f1e7e48)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.788921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:44.875199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1008/1973] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:44 2022] POST /identity/v3/users => generated 293 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:44.878648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:44.892159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94690d7d-1bdb-4753-ae6e-ada6248d417a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:44.904485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:44.904485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.904485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.904485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.904827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.904827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:44.920567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1009/1974] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:44 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:44.921551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:44.933000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962f0acb-80c9-40bc-91c9-7baa8dd04c58 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:44.938108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:44.938897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:44.939236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:44.939681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] PATH_INFO: `/v3/projects/41f95b0165414ef49a5b29ff4177ca10/users/80dd504e252b461c8b50cd8134bb9168/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:44.952781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:44.957221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=41f95b0165414ef49a5b29ff4177ca10, user_id=80dd504e252b461c8b50cd8134bb9168, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:44.960664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:45.034177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '209ef52a4a7b4378a3496e370f1e7e48'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.043125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '209ef52a4a7b4378a3496e370f1e7e48'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.043870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '209ef52a4a7b4378a3496e370f1e7e48'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.044534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '209ef52a4a7b4378a3496e370f1e7e48'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.049858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:45.063616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1010/1975] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:44 2022] PUT /identity/v3/projects/41f95b0165414ef49a5b29ff4177ca10/users/80dd504e252b461c8b50cd8134bb9168/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:45.067229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:45.069667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:45.160047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd3b6ea-c03e-4414-ba95-3d3dcd1927c1 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:45.167295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:45.167925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:45.168211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:45.168477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:45.169120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:45.170843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:45.182339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1011/1976] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:45 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:45.186250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:45.196846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-68f61758-7cb1-43f1-84d5-c43d68f33104 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '209ef52a4a7b4378a3496e370f1e7e48'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.199568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07812147-c776-4a1e-918b-350201279e36 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:45.205907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:45.206702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:45.206923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:45.207215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] PATH_INFO: `/v3/projects/41f95b0165414ef49a5b29ff4177ca10/users/80dd504e252b461c8b50cd8134bb9168/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:45.209133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] RBAC: Authorizing `identity:create_grant(project_id=41f95b0165414ef49a5b29ff4177ca10, user_id=80dd504e252b461c8b50cd8134bb9168, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:45.213700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:45.227340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1012/1977] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:45 2022] PUT /identity/v3/projects/41f95b0165414ef49a5b29ff4177ca10/users/80dd504e252b461c8b50cd8134bb9168/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:45.262109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 966/1978] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:44 2022] DELETE /identity/v3/users/209ef52a4a7b4378a3496e370f1e7e48 => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:45.288652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:45.290721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:45.369720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d221d55-369f-41e8-9b2c-b151723d7414 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:45.446394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:45.447122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:45.447448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:45.447670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] PATH_INFO: `/v3/users/0ee689c80ad34b189e6d87c4cb7fb1c7` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:45.449001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ee689c80ad34b189e6d87c4cb7fb1c7)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:45.451010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:45.671734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ee689c80ad34b189e6d87c4cb7fb1c7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.687526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ee689c80ad34b189e6d87c4cb7fb1c7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.688382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ee689c80ad34b189e6d87c4cb7fb1c7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.689073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ee689c80ad34b189e6d87c4cb7fb1c7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.745225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:45.774193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ee689c80ad34b189e6d87c4cb7fb1c7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:45.798568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1013/1979] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:45 2022] DELETE /identity/v3/users/0ee689c80ad34b189e6d87c4cb7fb1c7 => generated 0 bytes in 507 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:45.800176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:45.800679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:45.800772 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:45.821073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:45.822659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:45.835704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:45.903302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 967/1980] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:45.916289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:45.922555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:45.983052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1485fa-37dd-4cc3-9868-3613b4d96809 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:45.986856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:45.987472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:45.987755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:45.988028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:45.999462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:46.000999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:46.013910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1014/1981] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:46.172777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:46.174455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:46.187313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64999e00-0df6-4c09-a2bd-34bf601b8152 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:46.191942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:46.192560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:46.192835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:46.193102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] PATH_INFO: `/v3/projects/383afa70a41e4056a6275d6058ec5e30` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:46.195058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] RBAC: Authorizing `identity:delete_project(project_id=383afa70a41e4056a6275d6058ec5e30)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:46.196968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:46.237444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:46.296207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:46.326536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '383afa70a41e4056a6275d6058ec5e30', 'request_id': 'req-81a2fa7c-8431-4db9-8db7-18e1a291a472'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.403094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '383afa70a41e4056a6275d6058ec5e30', 'request_id': 'req-81a2fa7c-8431-4db9-8db7-18e1a291a472'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.411154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '383afa70a41e4056a6275d6058ec5e30', 'request_id': 'req-81a2fa7c-8431-4db9-8db7-18e1a291a472'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.425952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] The token cache is being invalidate because project 383afa70a41e4056a6275d6058ec5e30 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:46.426244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.426931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.428466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 968/1982] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:46 2022] DELETE /identity/v3/projects/383afa70a41e4056a6275d6058ec5e30 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:46.435474 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:46.446753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281f8aac-6b05-4cb1-8936-3661286c0bbf service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:46.515603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:46.516746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:46.517397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:46.517803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] PATH_INFO: `/v3/users/a10d1d17097d4233968634079b0a31d0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:46.519140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a10d1d17097d4233968634079b0a31d0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:46.521549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:46.710791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:46.713628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:46.746629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2fa7c-8431-4db9-8db7-18e1a291a472 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:46.754196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:46.754196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:46.754196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:46.754614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] PATH_INFO: `/v3/projects/35822fc8958f43d997f535fcb6a5aea6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:46.759581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] RBAC: Authorizing `identity:delete_project(project_id=35822fc8958f43d997f535fcb6a5aea6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:46.762104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:46.777299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a10d1d17097d4233968634079b0a31d0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.785981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a10d1d17097d4233968634079b0a31d0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.788818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a10d1d17097d4233968634079b0a31d0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.788818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a10d1d17097d4233968634079b0a31d0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.907161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06411da3-969e-41e3-80cc-99f8c8c7be09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a10d1d17097d4233968634079b0a31d0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:46.938961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1015/1983] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:46 2022] DELETE /identity/v3/users/a10d1d17097d4233968634079b0a31d0 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:46.971574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:46.971918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:46.972539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:46.989131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35822fc8958f43d997f535fcb6a5aea6', 'request_id': 'req-e266e4eb-80c8-4104-90df-317f1090027b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.019953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] MFA Rules not processed for user `80dd504e252b461c8b50cd8134bb9168`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:47.021593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:47.085939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35822fc8958f43d997f535fcb6a5aea6', 'request_id': 'req-e266e4eb-80c8-4104-90df-317f1090027b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.099458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35822fc8958f43d997f535fcb6a5aea6', 'request_id': 'req-e266e4eb-80c8-4104-90df-317f1090027b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.117170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] The token cache is being invalidate because project 35822fc8958f43d997f535fcb6a5aea6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:47.119502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.119502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.131350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 969/1984] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:46 2022] DELETE /identity/v3/projects/35822fc8958f43d997f535fcb6a5aea6 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:47.132536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:47.139085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:47.151787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1016/1985] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:46 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:47.167811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:47.171316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:47.235044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e266e4eb-80c8-4104-90df-317f1090027b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:47.242335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:47.243171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:47.244036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:47.244445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] PATH_INFO: `/v3/users/02f30723ae9e4ad99e2a751d3fc8704f` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:47.246929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02f30723ae9e4ad99e2a751d3fc8704f)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:47.250622 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:47.260264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35d1b65-37b9-4c6a-b3cd-970a1fbaed95 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:47.275088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dcf1a7-fc4f-4665-ad06-3a132fa0ac9f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:47.278526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcf1a7-fc4f-4665-ad06-3a132fa0ac9f service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:47.279206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcf1a7-fc4f-4665-ad06-3a132fa0ac9f service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:47.279705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcf1a7-fc4f-4665-ad06-3a132fa0ac9f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:47.285914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-00dcf1a7-fc4f-4665-ad06-3a132fa0ac9f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:47.367522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dcf1a7-fc4f-4665-ad06-3a132fa0ac9f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:47.369132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dcf1a7-fc4f-4665-ad06-3a132fa0ac9f service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:47.397619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1017/1986] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:47 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:47.520859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cad026-43ee-457e-9eb3-c2d5060f848d None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:47.521908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cad026-43ee-457e-9eb3-c2d5060f848d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:47.522231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cad026-43ee-457e-9eb3-c2d5060f848d None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:47.524084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1018/1987] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:52:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:52:47.548699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02f30723ae9e4ad99e2a751d3fc8704f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.557723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:47.558162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:47.558469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:47.562761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02f30723ae9e4ad99e2a751d3fc8704f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.563459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02f30723ae9e4ad99e2a751d3fc8704f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.564063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02f30723ae9e4ad99e2a751d3fc8704f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.585040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:47.586287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:47.687399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1019/1988] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:52:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:47.688444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:47.689193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02f30723ae9e4ad99e2a751d3fc8704f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.690516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:47.711750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f4580c-b171-4674-aed5-0a24317372bd None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:47.717636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:47.718577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:47.719069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:47.719539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] PATH_INFO: `/v3/projects/75817b1bc85049b3bb2be199afc53e8f` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:47.730846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 970/1989] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:47 2022] DELETE /identity/v3/users/02f30723ae9e4ad99e2a751d3fc8704f => generated 0 bytes in 593 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:47.738223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75817b1bc85049b3bb2be199afc53e8f)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:47.742328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:47.898761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75817b1bc85049b3bb2be199afc53e8f', 'request_id': 'req-583003a9-1e85-4458-a8c3-c502e0cdf593'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.900253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:47.901976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:47.906631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75817b1bc85049b3bb2be199afc53e8f', 'request_id': 'req-583003a9-1e85-4458-a8c3-c502e0cdf593'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.974310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75817b1bc85049b3bb2be199afc53e8f', 'request_id': 'req-583003a9-1e85-4458-a8c3-c502e0cdf593'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.993105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] The token cache is being invalidate because project 75817b1bc85049b3bb2be199afc53e8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:47.993403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.994091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-583003a9-1e85-4458-a8c3-c502e0cdf593 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:47.995811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1020/1990] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:47 2022] DELETE /identity/v3/projects/75817b1bc85049b3bb2be199afc53e8f => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:48.002318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa376b75-aeb0-4222-80c9-f57d986d2d47 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.002916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa376b75-aeb0-4222-80c9-f57d986d2d47 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.003595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa376b75-aeb0-4222-80c9-f57d986d2d47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.008544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975bbdd2-3500-4380-98da-17c8ecc51234 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.057436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-fa376b75-aeb0-4222-80c9-f57d986d2d47 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:48.060196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fa376b75-aeb0-4222-80c9-f57d986d2d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.099782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08d4bc7-852c-407f-b487-81d40275223c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.102246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08d4bc7-852c-407f-b487-81d40275223c service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.102786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08d4bc7-852c-407f-b487-81d40275223c service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.103198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08d4bc7-852c-407f-b487-81d40275223c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.105347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c08d4bc7-852c-407f-b487-81d40275223c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.118719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08d4bc7-852c-407f-b487-81d40275223c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.120467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08d4bc7-852c-407f-b487-81d40275223c service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.144893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 971/1991] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:48.172336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1021/1992] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:48.181844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd809129-a5dd-4064-9ec5-2ee1f0925cdc None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.182508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd809129-a5dd-4064-9ec5-2ee1f0925cdc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.182508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd809129-a5dd-4064-9ec5-2ee1f0925cdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.207074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-dd809129-a5dd-4064-9ec5-2ee1f0925cdc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:48.208448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-dd809129-a5dd-4064-9ec5-2ee1f0925cdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.223179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 972/1993] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:48.224584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa376b75-aeb0-4222-80c9-f57d986d2d47 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.237252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa376b75-aeb0-4222-80c9-f57d986d2d47 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.243702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.246287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.246287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.246287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.246287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:48.246287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.248684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.272893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1022/1994] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:48.277851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd809129-a5dd-4064-9ec5-2ee1f0925cdc None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.288146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd809129-a5dd-4064-9ec5-2ee1f0925cdc None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.293412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.294114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.294495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.294899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.295608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.297555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.382340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:48.394038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.396200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.409903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1878b545-ac0e-40af-860a-32174bdc7ac9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.427110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.429240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.429732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.430611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] PATH_INFO: `/v3/projects/ea594acdd47243128163b58b12761733` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.432074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea594acdd47243128163b58b12761733)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.434240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.444315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 973/1995] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/projects => generated 375 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:48.445532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.512293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e960eec-0d27-4de2-b9e0-a336f08d936d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.517128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.517746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.518024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.518295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.518949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.520938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.539914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:48.543111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea594acdd47243128163b58b12761733', 'request_id': 'req-c4206f4f-e31a-4971-a683-b0514d61a735'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:48.550224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea594acdd47243128163b58b12761733', 'request_id': 'req-c4206f4f-e31a-4971-a683-b0514d61a735'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:48.585951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 974/1996] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/users => generated 288 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:48.590161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.600589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fa2b4c-521b-4eca-bfcb-9b7eac73f163 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.602351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea594acdd47243128163b58b12761733', 'request_id': 'req-c4206f4f-e31a-4971-a683-b0514d61a735'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:48.620449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] The token cache is being invalidate because project ea594acdd47243128163b58b12761733 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:48.620741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:48.621421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c4206f4f-e31a-4971-a683-b0514d61a735 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:48.626384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1023/1997] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:48 2022] DELETE /identity/v3/projects/ea594acdd47243128163b58b12761733 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:48.627987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7ec12-6b08-4d51-9cd4-54b1b48a03ca None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.628378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7ec12-6b08-4d51-9cd4-54b1b48a03ca None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.629028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7ec12-6b08-4d51-9cd4-54b1b48a03ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.653012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-fbc7ec12-6b08-4d51-9cd4-54b1b48a03ca None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:48.655167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fbc7ec12-6b08-4d51-9cd4-54b1b48a03ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.665760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.667593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.667593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.667593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.667593 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.669147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.681847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 975/1998] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:48 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:48.685849 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.687625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.700279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbbe510-cc12-41a1-8948-5ced94e0e2a4 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.705408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.706007 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.706285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.706603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.707238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.709010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.721297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 976/1999] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:48 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:48.726383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.727480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1024/2000] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:48.734896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a79534d-dfd8-4974-a961-55db8e55447c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.735187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a79534d-dfd8-4974-a961-55db8e55447c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.735463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a79534d-dfd8-4974-a961-55db8e55447c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.736018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14c3b3-c963-4e7c-8378-914d252d17b0 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.741228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177f98b5-06d2-4e77-ad1d-651cac8a2b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.742227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f98b5-06d2-4e77-ad1d-651cac8a2b35 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.742538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f98b5-06d2-4e77-ad1d-651cac8a2b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.742905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f98b5-06d2-4e77-ad1d-651cac8a2b35 admin admin] PATH_INFO: `/v3/projects/4762ef4c48804923bc556537c5ee651c/users/8e73f816524a41b0928bded054e5b128/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.758632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8a79534d-dfd8-4974-a961-55db8e55447c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:48.759091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177f98b5-06d2-4e77-ad1d-651cac8a2b35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4762ef4c48804923bc556537c5ee651c, user_id=8e73f816524a41b0928bded054e5b128, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.759905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8a79534d-dfd8-4974-a961-55db8e55447c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.762549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177f98b5-06d2-4e77-ad1d-651cac8a2b35 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.767916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1025/2001] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:48.772597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a79534d-dfd8-4974-a961-55db8e55447c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.781636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a79534d-dfd8-4974-a961-55db8e55447c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.786296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.786912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.787191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.788112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.788927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:48.789348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.790967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.806964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1026/2002] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:48.807948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.817995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4007ded-039c-4eb6-b964-7f2a4728aa39 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:48.823690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:48.824539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.824821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.825095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.825729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:48.827772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:48.843102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 977/2003] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:48 2022] PUT /identity/v3/projects/4762ef4c48804923bc556537c5ee651c/users/8e73f816524a41b0928bded054e5b128/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:48.845101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:48.845496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:48.845877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:48.876948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] MFA Rules not processed for user `8e73f816524a41b0928bded054e5b128`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:48.878203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:48.925577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1027/2004] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/projects => generated 377 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:48.940720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.953275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 978/2005] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:48.965180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:48.966697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:49.043064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918da51c-b898-490d-bade-2530b05f42fb admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:49.049120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:49.049828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.050181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.050575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.051333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:49.054912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:49.098170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd6a4f-7801-41cd-8e64-ce7119835bd1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:49.163656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:49.164528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.164896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.165572 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.212341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1028/2006] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:48 2022] POST /identity/v3/users => generated 289 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:49.215820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:49.222812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:49.243362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:49.245386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:49.256948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 979/2007] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:48 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:49.258323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:49.272707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e245b-e26c-4b15-a9d0-e0ad1834e60c service glance] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:49.279242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:49.279242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.279242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.279651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.280126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:49.281805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:49.307712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 980/2008] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:49 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:49.308695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:49.321615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280efb5b-a022-43d5-8a44-359697dfc75f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:49.334078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20508ea9-922e-4336-8fe9-ff41d4cb6ea8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:49.338143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:49.339709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.339998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.340273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.340926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:49.344858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:49.347028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27297294-f7f2-48a8-a020-f0b97a2b45cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:49.347834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27297294-f7f2-48a8-a020-f0b97a2b45cf service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.348121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27297294-f7f2-48a8-a020-f0b97a2b45cf service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.348394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27297294-f7f2-48a8-a020-f0b97a2b45cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.352332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-27297294-f7f2-48a8-a020-f0b97a2b45cf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:49.368150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 981/2009] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:49 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:49.378788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:49.397254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dfa779-258a-4823-b384-6f88c66a5260 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:49.405083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:49.405814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.406081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.406385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] PATH_INFO: `/v3/projects/303be2948a6f4834a7221f47b03635b5/users/5a9687ba31f246408d0e72d4f6d6799e/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.428896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=303be2948a6f4834a7221f47b03635b5, user_id=5a9687ba31f246408d0e72d4f6d6799e, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:49.431508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:49.464328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27297294-f7f2-48a8-a020-f0b97a2b45cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:49.466669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27297294-f7f2-48a8-a020-f0b97a2b45cf service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:49.480196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1029/2010] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:49 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:49.562260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 982/2011] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:49 2022] PUT /identity/v3/projects/303be2948a6f4834a7221f47b03635b5/users/5a9687ba31f246408d0e72d4f6d6799e/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:49.564333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.564632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.565333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.617937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] MFA Rules not processed for user `5a9687ba31f246408d0e72d4f6d6799e`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:49.619421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:49.700983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1030/2012] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:49.754076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:49.813351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8c22a5-4126-4780-aad8-7a33ee4e38e8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:49.817376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b93d9c-a00f-4a9a-b2d0-2f23d16e67fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:49.818002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b93d9c-a00f-4a9a-b2d0-2f23d16e67fd service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:49.818280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b93d9c-a00f-4a9a-b2d0-2f23d16e67fd service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:49.818776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b93d9c-a00f-4a9a-b2d0-2f23d16e67fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:49.828445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b93d9c-a00f-4a9a-b2d0-2f23d16e67fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:49.830789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b93d9c-a00f-4a9a-b2d0-2f23d16e67fd service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:49.839935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 983/2013] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:49 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:50.274728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:50.379411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:52.212637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:52.223359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:52.335601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe3033a-ab48-4657-b8b3-dd7d74b9d7b9 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:52.342447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:52.343106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:52.343408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:52.343683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] PATH_INFO: `/v3/users/17a61a5c1ec348d196890c40fc11bf3b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:52.345323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=17a61a5c1ec348d196890c40fc11bf3b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:52.347660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:52.609058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17a61a5c1ec348d196890c40fc11bf3b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:52.616062 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17a61a5c1ec348d196890c40fc11bf3b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:52.616745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17a61a5c1ec348d196890c40fc11bf3b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:52.617408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17a61a5c1ec348d196890c40fc11bf3b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:52.729289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17a61a5c1ec348d196890c40fc11bf3b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:52.756992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1031/2014] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:52 2022] DELETE /identity/v3/users/17a61a5c1ec348d196890c40fc11bf3b => generated 0 bytes in 545 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:52.767053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:52.767053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:52.767053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:52.785672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:52.786958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:52.869308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 984/2015] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:52.888501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:52.888501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:52.947264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d07fb23-9488-4c28-bfd0-41281e1447e1 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:52.950787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:52.951870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:52.951870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:52.952103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:52.961574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:52.967974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:52.979175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1032/2016] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:52.982091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:53.410569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:53.613826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:53.613826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:53.632528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdda90e-c393-43fe-980a-bcfc74776147 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:53.642912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:53.642912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:53.642912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:53.642912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] PATH_INFO: `/v3/projects/6be9611496034f7d981daac6762ac33f` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:53.643246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6be9611496034f7d981daac6762ac33f)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:53.645706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:53.796441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6be9611496034f7d981daac6762ac33f', 'request_id': 'req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:53.912031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6be9611496034f7d981daac6762ac33f', 'request_id': 'req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:53.920435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6be9611496034f7d981daac6762ac33f', 'request_id': 'req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:53.937773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] The token cache is being invalidate because project 6be9611496034f7d981daac6762ac33f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:53.938055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:53.938921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:53.941125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 985/2017] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:53 2022] DELETE /identity/v3/projects/6be9611496034f7d981daac6762ac33f => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:53.944892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:53.960239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f678a8b0-2336-4a48-a72b-712a8b28fb91 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:54.031133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:54.032043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:54.032325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:54.032730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] PATH_INFO: `/v3/users/6d02b613b554431bac7429a0edef88d5` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:54.034462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d02b613b554431bac7429a0edef88d5)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:54.036708 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:54.218382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d02b613b554431bac7429a0edef88d5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:54.249433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d02b613b554431bac7429a0edef88d5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:54.250875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d02b613b554431bac7429a0edef88d5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:54.252171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d02b613b554431bac7429a0edef88d5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:54.289330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-905892af-f5ef-425d-96d9-2e3c067f8274 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d02b613b554431bac7429a0edef88d5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:54.313069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1033/2018] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:52:53 2022] DELETE /identity/v3/users/6d02b613b554431bac7429a0edef88d5 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:54.489109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:54.751764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:54.753288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:54.765608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:54.769817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1acadc-3de5-42a8-9dc8-58f6743cb9d3 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:54.841650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:54.842359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:54.842591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:54.842899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] PATH_INFO: `/v3/projects/a43da02339fb4b238fccc578924ab426` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:54.844073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a43da02339fb4b238fccc578924ab426)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:54.845965 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:54.987669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a43da02339fb4b238fccc578924ab426', 'request_id': 'req-f9196a3a-9a45-406c-aaf9-7395603d3764'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:55.105730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a43da02339fb4b238fccc578924ab426', 'request_id': 'req-f9196a3a-9a45-406c-aaf9-7395603d3764'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:55.113533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a43da02339fb4b238fccc578924ab426', 'request_id': 'req-f9196a3a-9a45-406c-aaf9-7395603d3764'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:55.125409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] The token cache is being invalidate because project a43da02339fb4b238fccc578924ab426 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:52:55.125729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:55.126361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9196a3a-9a45-406c-aaf9-7395603d3764 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:52:55.129109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 986/2019] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:52:54 2022] DELETE /identity/v3/projects/a43da02339fb4b238fccc578924ab426 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:55.134839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cf8de4-604b-4939-bd6f-36309fdd9376 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.139220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cf8de4-604b-4939-bd6f-36309fdd9376 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.139220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cf8de4-604b-4939-bd6f-36309fdd9376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.166437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-69cf8de4-604b-4939-bd6f-36309fdd9376 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:55.169367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-69cf8de4-604b-4939-bd6f-36309fdd9376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:55.238665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1034/2020] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:55.248116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844d114d-0d30-4d8a-9175-6e0b283e9132 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.258365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844d114d-0d30-4d8a-9175-6e0b283e9132 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.258365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844d114d-0d30-4d8a-9175-6e0b283e9132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.270145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-844d114d-0d30-4d8a-9175-6e0b283e9132 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:55.271462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-844d114d-0d30-4d8a-9175-6e0b283e9132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:55.286979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:55.287412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cf8de4-604b-4939-bd6f-36309fdd9376 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:55.287616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 987/2021] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:55.300820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cf8de4-604b-4939-bd6f-36309fdd9376 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:55.307650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:55.308370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.308721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.309066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.309889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:52:55.310170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:55.312805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:55.329753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1035/2022] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:52:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:52:55.331363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844d114d-0d30-4d8a-9175-6e0b283e9132 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:55.348340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844d114d-0d30-4d8a-9175-6e0b283e9132 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:55.354082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:55.355031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.355322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.355596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.356791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:55.360045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:55.473109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 988/2023] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:55 2022] POST /identity/v3/projects => generated 385 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:55.477207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:55.487484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fe493b-b978-4099-bcae-978636781396 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:55.547047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:55.547849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.549050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.549351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.549999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:55.551949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:55.664331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1036/2024] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:55 2022] POST /identity/v3/users => generated 293 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:55.666811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:55.678161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8eb8b0-1af3-4ec6-9c05-fe8ae04f6262 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:55.682753 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:55.683530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.683871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.684204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.684915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:55.686769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:55.697926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 989/2025] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:55.702720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:55.717171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db837448-adf3-401f-b362-ad469d561abb admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:55.722752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72882975-6d8c-4ff2-adf2-e6e1c14058a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:55.723482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72882975-6d8c-4ff2-adf2-e6e1c14058a3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.724005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72882975-6d8c-4ff2-adf2-e6e1c14058a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.724105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72882975-6d8c-4ff2-adf2-e6e1c14058a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.724752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72882975-6d8c-4ff2-adf2-e6e1c14058a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:55.726543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72882975-6d8c-4ff2-adf2-e6e1c14058a3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:55.745360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1037/2026] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:55 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:52:55.746841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:55.759776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97a46b-4416-44fa-9531-42eb4e656fba admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:55.765251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:55.765986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.766342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.766754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] PATH_INFO: `/v3/projects/98c4346c134348c0ba2fd84a5cb1600b/users/5bbbb03a03c14d509c1b6bc4f9f978b1/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.782245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] RBAC: Authorizing `identity:create_grant(project_id=98c4346c134348c0ba2fd84a5cb1600b, user_id=5bbbb03a03c14d509c1b6bc4f9f978b1, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:55.784990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:55.877309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 990/2027] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:55 2022] PUT /identity/v3/projects/98c4346c134348c0ba2fd84a5cb1600b/users/5bbbb03a03c14d509c1b6bc4f9f978b1/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:55.882814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:55.888477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a32db78-59db-40af-9eb4-349041c947e1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:55.888759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a32db78-59db-40af-9eb4-349041c947e1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:55.904213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a32db78-59db-40af-9eb4-349041c947e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:55.918554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9a32db78-59db-40af-9eb4-349041c947e1 None None] MFA Rules not processed for user `5bbbb03a03c14d509c1b6bc4f9f978b1`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:55.919723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9a32db78-59db-40af-9eb4-349041c947e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:55.985881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1038/2028] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:52:55.990577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:56.012988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:56.059536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921552d-916d-4f12-bd30-0bcc436c826e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:56.065640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:56.066345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.066770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.067120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.067860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:56.069908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:56.147195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 991/2029] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:52:55 2022] POST /identity/v3/projects => generated 383 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:52:56.152156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a32db78-59db-40af-9eb4-349041c947e1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:56.163832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a32db78-59db-40af-9eb4-349041c947e1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:56.227831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:56.229656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.229957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.230236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.230916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:56.232877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:56.287393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:56.324674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1039/2030] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:52:56 2022] POST /identity/v3/users => generated 298 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:52:56.330000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:56.343967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a88cb-016c-4005-9a06-51737294cc03 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:56.350954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:56.352259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.352728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.356917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.356917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:56.356917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:56.370664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 992/2031] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:56 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:56.379186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:56.396097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5faae6-3bd9-4d05-b114-a3ff3537c52c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:56.402101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:56.402843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.403121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.403393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] PATH_INFO: `/v3/projects/a502a04efa464890a297b3b9257da948/users/eaf0a9f154864610807aa3aafaa3d830/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.420505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=a502a04efa464890a297b3b9257da948, user_id=eaf0a9f154864610807aa3aafaa3d830, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:56.423518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:56.540227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1040/2032] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:56 2022] PUT /identity/v3/projects/a502a04efa464890a297b3b9257da948/users/eaf0a9f154864610807aa3aafaa3d830/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:52:56.544779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:56.557118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085f1136-5736-494f-9373-91c3609ececb admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:56.635855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e61a06-ab4d-4700-af86-b69886157768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:56.636472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e61a06-ab4d-4700-af86-b69886157768 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.636751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e61a06-ab4d-4700-af86-b69886157768 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.637025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e61a06-ab4d-4700-af86-b69886157768 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.637671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e61a06-ab4d-4700-af86-b69886157768 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:56.639293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e61a06-ab4d-4700-af86-b69886157768 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:56.650591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 993/2033] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:52:56 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:52:56.655153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:56.665563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be16a41-0308-4b14-85bf-b72c35a1ecdc admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:56.670770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:56.671360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.671636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.671908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] PATH_INFO: `/v3/projects/a502a04efa464890a297b3b9257da948/users/eaf0a9f154864610807aa3aafaa3d830/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.673801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a502a04efa464890a297b3b9257da948, user_id=eaf0a9f154864610807aa3aafaa3d830, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:56.677052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:56.678075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:56.689541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1041/2034] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:52:56 2022] PUT /identity/v3/projects/a502a04efa464890a297b3b9257da948/users/eaf0a9f154864610807aa3aafaa3d830/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:52:56.700503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.700780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.701077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.732856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] MFA Rules not processed for user `eaf0a9f154864610807aa3aafaa3d830`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:52:56.734093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:56.812451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 994/2035] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:52:56 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:52:56.824281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:56.826186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:56.877454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:56.895895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58428938-1c08-47a2-8464-782d2075d73e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:56.903500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:56.904458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:56.904735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:56.905040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:56.971013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:56.972636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:56.982145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1042/2036] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:57.072273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:57.185541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:57.484080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:57.858388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:57.946295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:58.135677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:58.420154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:58.596874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:58.664522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:58.865570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:59.020244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:06.061631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:06.063743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:06.080119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aedca2a-b494-4521-a498-699e7c6edeb6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:06.084879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d395b67-4cf1-4aa7-aa16-d92376091617 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:06.085812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d395b67-4cf1-4aa7-aa16-d92376091617 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:06.086471 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d395b67-4cf1-4aa7-aa16-d92376091617 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:06.087014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d395b67-4cf1-4aa7-aa16-d92376091617 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:06.099059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d395b67-4cf1-4aa7-aa16-d92376091617 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:06.100820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d395b67-4cf1-4aa7-aa16-d92376091617 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:06.113322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 995/2037] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:06 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:09.065451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=65,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:15.389724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:15.480515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128e8b9-7691-40eb-b9f7-93aab59cc642 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:15.485112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:15.485758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:15.486034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:15.486306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] PATH_INFO: `/v3/users/5bbbb03a03c14d509c1b6bc4f9f978b1` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:15.487490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bbbb03a03c14d509c1b6bc4f9f978b1)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:15.489407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:15.654315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bbbb03a03c14d509c1b6bc4f9f978b1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:15.662258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bbbb03a03c14d509c1b6bc4f9f978b1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:15.663105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bbbb03a03c14d509c1b6bc4f9f978b1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:15.663854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bbbb03a03c14d509c1b6bc4f9f978b1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:15.754701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bbbb03a03c14d509c1b6bc4f9f978b1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:15.775199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1043/2038] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:15 2022] DELETE /identity/v3/users/5bbbb03a03c14d509c1b6bc4f9f978b1 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:15.779751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:15.780212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:15.780302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:15.801827 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:15.803118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:15.828470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:15.887329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 996/2039] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:15.899087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:15.901442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:15.984829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d6654-51eb-4622-aa0c-0c4497f4f0fa admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:15.988780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:15.989722 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:15.990000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:15.990664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:16.000822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:16.002312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:16.013195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1044/2040] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:16.491798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:16.494050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:16.511028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d95191-c6f9-4c0b-bc25-6bcf047f71b5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:16.515889 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:16.516548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:16.516825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:16.517269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] PATH_INFO: `/v3/projects/98c4346c134348c0ba2fd84a5cb1600b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:16.518618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] RBAC: Authorizing `identity:delete_project(project_id=98c4346c134348c0ba2fd84a5cb1600b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:16.520536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:16.678925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98c4346c134348c0ba2fd84a5cb1600b', 'request_id': 'req-b8c94070-6827-46e9-b000-530578bb4775'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:16.754983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98c4346c134348c0ba2fd84a5cb1600b', 'request_id': 'req-b8c94070-6827-46e9-b000-530578bb4775'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:16.764934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98c4346c134348c0ba2fd84a5cb1600b', 'request_id': 'req-b8c94070-6827-46e9-b000-530578bb4775'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:16.776458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] The token cache is being invalidate because project 98c4346c134348c0ba2fd84a5cb1600b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:16.776960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:16.777639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:16.779302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 997/2041] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:16 2022] DELETE /identity/v3/projects/98c4346c134348c0ba2fd84a5cb1600b => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:16.784226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:16.795837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef48e254-7a45-4d7d-9d55-bdb4a9dc8dab service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:16.811132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:16.812835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:16.819041 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:16.860995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:16.861760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:16.862021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:16.862482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] PATH_INFO: `/v3/users/eaf0a9f154864610807aa3aafaa3d830` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:16.863633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eaf0a9f154864610807aa3aafaa3d830)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:16.865592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:16.883214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c94070-6827-46e9-b000-530578bb4775 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:16.888829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:16.889604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:16.889887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:16.890155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] PATH_INFO: `/v3/users/5a9687ba31f246408d0e72d4f6d6799e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:16.891662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a9687ba31f246408d0e72d4f6d6799e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:16.893249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:17.095117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eaf0a9f154864610807aa3aafaa3d830'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.103916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eaf0a9f154864610807aa3aafaa3d830'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.104634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eaf0a9f154864610807aa3aafaa3d830'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.105253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eaf0a9f154864610807aa3aafaa3d830'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.187483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a9687ba31f246408d0e72d4f6d6799e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.194004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:17.196865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a9687ba31f246408d0e72d4f6d6799e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.197712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a9687ba31f246408d0e72d4f6d6799e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.198342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a9687ba31f246408d0e72d4f6d6799e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.219617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4c25e4c9-da33-4d29-95f2-d2a059f6d518 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eaf0a9f154864610807aa3aafaa3d830'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.237653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1045/2042] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:16 2022] DELETE /identity/v3/users/eaf0a9f154864610807aa3aafaa3d830 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:17.294865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a9687ba31f246408d0e72d4f6d6799e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:17.314866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 998/2043] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:16 2022] DELETE /identity/v3/users/5a9687ba31f246408d0e72d4f6d6799e => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:17.319308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:17.319589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:17.319858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:17.340843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:17.342113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:17.407581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1046/2044] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:17.417902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:17.419453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:17.518969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5474b248-6df9-4c87-91bc-8e686d29943a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:17.521919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e260c6-09b4-4e3d-801e-d22511bc20a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:17.522821 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e260c6-09b4-4e3d-801e-d22511bc20a9 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:17.523104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e260c6-09b4-4e3d-801e-d22511bc20a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:17.523371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e260c6-09b4-4e3d-801e-d22511bc20a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:17.533912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e260c6-09b4-4e3d-801e-d22511bc20a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:17.535582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e260c6-09b4-4e3d-801e-d22511bc20a9 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:17.550591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 999/2045] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:17.839143 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:17.841330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:17.855503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f1d1f3-9bf9-4c6d-b03a-2a6e315ce5ee None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:17.861050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:17.861790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:17.862169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:17.862567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] PATH_INFO: `/v3/projects/a502a04efa464890a297b3b9257da948` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:17.863861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a502a04efa464890a297b3b9257da948)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:17.866522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:17.917072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21165811-af9e-405e-9ebb-6341ecd54258 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:17.917561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21165811-af9e-405e-9ebb-6341ecd54258 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:17.917649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21165811-af9e-405e-9ebb-6341ecd54258 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:17.919071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1000/2046] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:53:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:53:17.935812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:17.936286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:17.936755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:17.957150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] MFA Rules not processed for user `659e82e28da54e28a58d9d6e9c5e4215`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:17.958361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.010839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:18.022240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a502a04efa464890a297b3b9257da948', 'request_id': 'req-3abfa941-48a1-4057-aeb6-d4f76a631a46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.029295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a502a04efa464890a297b3b9257da948', 'request_id': 'req-3abfa941-48a1-4057-aeb6-d4f76a631a46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.053901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1001/2047] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:53:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:18.054779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.061828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.080338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1295dba4-f056-485e-8543-a71a98c0b2ff None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.095561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a502a04efa464890a297b3b9257da948', 'request_id': 'req-3abfa941-48a1-4057-aeb6-d4f76a631a46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.115649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] The token cache is being invalidate because project a502a04efa464890a297b3b9257da948 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:18.116459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.118871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.120850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1047/2048] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:17 2022] DELETE /identity/v3/projects/a502a04efa464890a297b3b9257da948 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:18.121643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.142070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.221551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.222498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.222871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.223249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] PATH_INFO: `/v3/projects/303be2948a6f4834a7221f47b03635b5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.224530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] RBAC: Authorizing `identity:delete_project(project_id=303be2948a6f4834a7221f47b03635b5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.226878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.244089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abfa941-48a1-4057-aeb6-d4f76a631a46 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.249814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475fa7b7-e358-4cf1-9d0b-6f26696a72ed service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.250752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475fa7b7-e358-4cf1-9d0b-6f26696a72ed service placement] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.251190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475fa7b7-e358-4cf1-9d0b-6f26696a72ed service placement] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.251616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475fa7b7-e358-4cf1-9d0b-6f26696a72ed service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.253991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:18.255844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-475fa7b7-e358-4cf1-9d0b-6f26696a72ed service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.270023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475fa7b7-e358-4cf1-9d0b-6f26696a72ed service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.276171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475fa7b7-e358-4cf1-9d0b-6f26696a72ed service placement] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.294606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1048/2049] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:18.296207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf601cf-ae4a-48d3-98ee-c9b1e5f37069 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.297537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf601cf-ae4a-48d3-98ee-c9b1e5f37069 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.297537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf601cf-ae4a-48d3-98ee-c9b1e5f37069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.325406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0bf601cf-ae4a-48d3-98ee-c9b1e5f37069 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:18.326727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0bf601cf-ae4a-48d3-98ee-c9b1e5f37069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.338147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1049/2050] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:18.346306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c7773-ef8a-44df-a660-de13c7456f22 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.346627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c7773-ef8a-44df-a660-de13c7456f22 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.346905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c7773-ef8a-44df-a660-de13c7456f22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.369848 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-643c7773-ef8a-44df-a660-de13c7456f22 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:18.371086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-643c7773-ef8a-44df-a660-de13c7456f22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.392171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1050/2051] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:18.397187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c7773-ef8a-44df-a660-de13c7456f22 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.407211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '303be2948a6f4834a7221f47b03635b5', 'request_id': 'req-87c714d0-5f9b-4e35-a8aa-fdd150d72971'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.408508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c7773-ef8a-44df-a660-de13c7456f22 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.464864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.465472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.465747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.466019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.466751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:18.467049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.468666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.479976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1051/2052] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:18.484624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.494440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f676e9-3177-4659-abfe-1473d1f64e71 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.499075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.499637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.499939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.500208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.500814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.502855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.527788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '303be2948a6f4834a7221f47b03635b5', 'request_id': 'req-87c714d0-5f9b-4e35-a8aa-fdd150d72971'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.535102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '303be2948a6f4834a7221f47b03635b5', 'request_id': 'req-87c714d0-5f9b-4e35-a8aa-fdd150d72971'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.545731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] The token cache is being invalidate because project 303be2948a6f4834a7221f47b03635b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:18.546022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.546788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-87c714d0-5f9b-4e35-a8aa-fdd150d72971 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:18.549408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1002/2053] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:18 2022] DELETE /identity/v3/projects/303be2948a6f4834a7221f47b03635b5 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:18.555059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cf6e4-8161-4c5e-9e86-17a385476c3c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.555344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cf6e4-8161-4c5e-9e86-17a385476c3c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.556139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cf6e4-8161-4c5e-9e86-17a385476c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.579155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ba1cf6e4-8161-4c5e-9e86-17a385476c3c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:18.580391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ba1cf6e4-8161-4c5e-9e86-17a385476c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.589303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1003/2054] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:18.596857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dce211-884b-4c5d-a826-edd87dc0bde9 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.597212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dce211-884b-4c5d-a826-edd87dc0bde9 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.597702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dce211-884b-4c5d-a826-edd87dc0bde9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.624438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e7dce211-884b-4c5d-a826-edd87dc0bde9 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:18.624677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1052/2055] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/projects => generated 383 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:18.626054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e7dce211-884b-4c5d-a826-edd87dc0bde9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.628460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.630878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:18.698673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92629ac4-27b9-43d6-ad27-7e3fa0e6ed23 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.704005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1004/2056] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:18.705148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dce211-884b-4c5d-a826-edd87dc0bde9 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.708315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.709080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.709458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.709821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.710593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.713279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.715091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dce211-884b-4c5d-a826-edd87dc0bde9 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.719885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.720537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.720816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.721082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.721764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:18.722016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.724075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.736926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1005/2057] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:18.748094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.752394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bb7d7-7f59-4811-86fb-e63b0ae56a20 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.758023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.759380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.759748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.760008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.760635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.762621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.829403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1053/2058] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/users => generated 292 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:18.833374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.843291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b2d56-e6ea-482a-a6aa-2c3482f7b79d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.848238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.848828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.849132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.849368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.850011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.851642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.865440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1054/2059] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:18 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:18.866344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.877240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dde6a48-55c5-4a83-b39a-65f89ff4f70d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.881825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:18.884337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.884929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.885205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.885471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.886150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.893747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.893857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1006/2060] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/projects => generated 375 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:18.894683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.901756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1055/2061] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:18.902822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:18.912294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bb2055-3bf0-4538-85eb-abe840b678e4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.964175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f46c35-af00-4dc0-8bf1-22bb17af02e9 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:18.973475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.977905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.977905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.978200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:18.978432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.978549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:18.978700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.978827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:18.978827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] PATH_INFO: `/v3/projects/00b0fcbba3a14cde9a5a5cfd0374d3c2/users/d86f06e7c1b64cc3826029b80cd6a360/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:18.978992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:18.994164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00b0fcbba3a14cde9a5a5cfd0374d3c2, user_id=d86f06e7c1b64cc3826029b80cd6a360, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:18.997227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.067194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1007/2062] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:18 2022] POST /identity/v3/users => generated 288 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:19.067194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:19.068209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.081088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3582601-d858-44e4-a664-d8884294b441 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:19.088498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:19.089502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.092102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.092102 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] PATH_INFO: `/v3/users/b0c2529a3b9e41e084c87c6be9ae22ab` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.092406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0c2529a3b9e41e084c87c6be9ae22ab)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:19.093340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1056/2063] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:18 2022] PUT /identity/v3/projects/00b0fcbba3a14cde9a5a5cfd0374d3c2/users/d86f06e7c1b64cc3826029b80cd6a360/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:19.094660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:19.095873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.164240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf99f86-ed3d-48bc-85b9-3b376953e9b8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:19.169742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f224fd0b-6fab-4f4a-892b-d4daed07b590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:19.170191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f224fd0b-6fab-4f4a-892b-d4daed07b590 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.170505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f224fd0b-6fab-4f4a-892b-d4daed07b590 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.170780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f224fd0b-6fab-4f4a-892b-d4daed07b590 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.171429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f224fd0b-6fab-4f4a-892b-d4daed07b590 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:19.173111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f224fd0b-6fab-4f4a-892b-d4daed07b590 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.183362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1057/2064] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:19 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:19.184599 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cca695-b6d4-410a-8765-86486827ae03 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.184884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cca695-b6d4-410a-8765-86486827ae03 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.185158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cca695-b6d4-410a-8765-86486827ae03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.206399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:19.212729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-35cca695-b6d4-410a-8765-86486827ae03 None None] MFA Rules not processed for user `d86f06e7c1b64cc3826029b80cd6a360`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:19.213936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-35cca695-b6d4-410a-8765-86486827ae03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.286683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1058/2065] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:19.287115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0c2529a3b9e41e084c87c6be9ae22ab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:19.287523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cca695-b6d4-410a-8765-86486827ae03 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:19.296378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0c2529a3b9e41e084c87c6be9ae22ab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:19.297737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0c2529a3b9e41e084c87c6be9ae22ab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:19.298739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0c2529a3b9e41e084c87c6be9ae22ab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:19.301634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cca695-b6d4-410a-8765-86486827ae03 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:19.310131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:19.310131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.310131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.310131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.310510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:19.310510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.320044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1059/2066] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:19.324024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:19.325877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.384012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0c2529a3b9e41e084c87c6be9ae22ab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:19.385503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226fb578-3981-4bff-ac4f-ca5f2a631301 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:19.389401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afad045-9b69-4958-b74e-7b50b43d517a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:19.390012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afad045-9b69-4958-b74e-7b50b43d517a service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.390290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afad045-9b69-4958-b74e-7b50b43d517a service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.393336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afad045-9b69-4958-b74e-7b50b43d517a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.393336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6afad045-9b69-4958-b74e-7b50b43d517a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.409642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afad045-9b69-4958-b74e-7b50b43d517a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:19.411900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afad045-9b69-4958-b74e-7b50b43d517a service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.419530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:19.429416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1008/2067] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:19 2022] DELETE /identity/v3/users/b0c2529a3b9e41e084c87c6be9ae22ab => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:19.430237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:19.435283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.436153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1060/2068] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:19 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:19.533485 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055f28d8-1f8b-4c3f-a012-0d1c5d7272f2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:19.541028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:19.542119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.542600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.543087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] PATH_INFO: `/v3/projects/242e1450de5b4b7b9cc030474995f021/users/c6082edc5c234fd1848b58b431c7cf4b/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.561523 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] RBAC: Authorizing `identity:create_grant(project_id=242e1450de5b4b7b9cc030474995f021, user_id=c6082edc5c234fd1848b58b431c7cf4b, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:19.565047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.654895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1009/2069] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:19 2022] PUT /identity/v3/projects/242e1450de5b4b7b9cc030474995f021/users/c6082edc5c234fd1848b58b431c7cf4b/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:19.665963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.666448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.666726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.694066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] MFA Rules not processed for user `c6082edc5c234fd1848b58b431c7cf4b`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:19.695302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.761730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:19.764069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.770702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1061/2070] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:19 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:19.844390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:INSERT=5,keystone:SELECT=85,keystone:DELETE=10 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:19.860813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:19.864003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:19.875366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b9fefb-f3b1-4fef-990e-3433eb36d25e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:19.880885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:19.881528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.881806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.882073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] PATH_INFO: `/v3/projects/894b70b0c4954b869c836a087285825a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.882393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:19.883232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=894b70b0c4954b869c836a087285825a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:19.885273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.925440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a15ce-087c-45f3-a5c4-bf1b7eb66a2b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:19.929022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:19.929620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:19.929897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:19.930165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:19.939525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:19.941087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:19.951593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1062/2071] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:19 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:19.983745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '894b70b0c4954b869c836a087285825a', 'request_id': 'req-e376fb33-7ce4-46b2-adda-be421104dbb4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.059613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '894b70b0c4954b869c836a087285825a', 'request_id': 'req-e376fb33-7ce4-46b2-adda-be421104dbb4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.067298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '894b70b0c4954b869c836a087285825a', 'request_id': 'req-e376fb33-7ce4-46b2-adda-be421104dbb4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.078948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] The token cache is being invalidate because project 894b70b0c4954b869c836a087285825a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:20.079230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.079890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.081382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1010/2072] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:19 2022] DELETE /identity/v3/projects/894b70b0c4954b869c836a087285825a => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:20.085389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:20.095968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a729dbe4-5e7a-4020-a054-2c6bf08aaf0a service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:20.163070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:20.163713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:20.163993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:20.164267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] PATH_INFO: `/v3/users/80dd504e252b461c8b50cd8134bb9168` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:20.165477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80dd504e252b461c8b50cd8134bb9168)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:20.167586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:20.346398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80dd504e252b461c8b50cd8134bb9168'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.354108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80dd504e252b461c8b50cd8134bb9168'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.354783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80dd504e252b461c8b50cd8134bb9168'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.355416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80dd504e252b461c8b50cd8134bb9168'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.448684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a67bba3d-0fb6-4d57-8988-4f8dee65fe11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80dd504e252b461c8b50cd8134bb9168'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:20.467723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1063/2073] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:20 2022] DELETE /identity/v3/users/80dd504e252b461c8b50cd8134bb9168 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:20.713943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:20.715494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:20.755851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e376fb33-7ce4-46b2-adda-be421104dbb4 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:20.828612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:20.829330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:20.829673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:20.830263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] PATH_INFO: `/v3/projects/41f95b0165414ef49a5b29ff4177ca10` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:20.832376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=41f95b0165414ef49a5b29ff4177ca10)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:20.834319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:20.999319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41f95b0165414ef49a5b29ff4177ca10', 'request_id': 'req-49b60367-4e04-49c8-9e7b-b7efd8217ae7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:21.086181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41f95b0165414ef49a5b29ff4177ca10', 'request_id': 'req-49b60367-4e04-49c8-9e7b-b7efd8217ae7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:21.096387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41f95b0165414ef49a5b29ff4177ca10', 'request_id': 'req-49b60367-4e04-49c8-9e7b-b7efd8217ae7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:21.108506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] The token cache is being invalidate because project 41f95b0165414ef49a5b29ff4177ca10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:21.108630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:21.109315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-49b60367-4e04-49c8-9e7b-b7efd8217ae7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:21.112319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1011/2074] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:20 2022] DELETE /identity/v3/projects/41f95b0165414ef49a5b29ff4177ca10 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:21.116085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df823584-52aa-4353-bfb6-c5d2e1b3a68b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.116584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df823584-52aa-4353-bfb6-c5d2e1b3a68b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.116869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df823584-52aa-4353-bfb6-c5d2e1b3a68b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.138631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-df823584-52aa-4353-bfb6-c5d2e1b3a68b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:21.139937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-df823584-52aa-4353-bfb6-c5d2e1b3a68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.172560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:21.202820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1064/2075] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:21.210616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585bd65-61a9-4b22-aeaa-539f36ca3330 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.210895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585bd65-61a9-4b22-aeaa-539f36ca3330 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.211164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585bd65-61a9-4b22-aeaa-539f36ca3330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.219910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:21.241941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8585bd65-61a9-4b22-aeaa-539f36ca3330 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:21.243848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8585bd65-61a9-4b22-aeaa-539f36ca3330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.256545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1012/2076] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:21.259042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df823584-52aa-4353-bfb6-c5d2e1b3a68b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.275079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df823584-52aa-4353-bfb6-c5d2e1b3a68b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.281774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.286362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.286362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.286362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.286362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:21.286362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.286362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.302782 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1065/2077] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:53:21.304160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585bd65-61a9-4b22-aeaa-539f36ca3330 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.314799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585bd65-61a9-4b22-aeaa-539f36ca3330 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.320733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.321701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.322159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.322585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.323530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.326253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.439382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1013/2078] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/projects => generated 385 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:21.440523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.450927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1935e-7650-4bf3-baf6-1952be716999 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.515268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.515882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.516214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.516489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.517146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.519166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.620295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1066/2079] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/users => generated 293 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:21.625577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.637224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1c874a-6a91-48ba-b317-e74c4c18bb81 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.642765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.654600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.654600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.654600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.654600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.654600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.658575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1014/2080] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:21.662762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.673863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092af672-13cf-40b6-bb3a-d233ac8a8118 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.678596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fad176-4e67-4617-b9d6-6d4d663c5b77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.679197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fad176-4e67-4617-b9d6-6d4d663c5b77 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.679670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fad176-4e67-4617-b9d6-6d4d663c5b77 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.679870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fad176-4e67-4617-b9d6-6d4d663c5b77 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.680711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fad176-4e67-4617-b9d6-6d4d663c5b77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.682378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fad176-4e67-4617-b9d6-6d4d663c5b77 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.693150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1067/2081] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:21.697657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.709165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e24b01-caa3-4a23-99cf-4392ee3f45e5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.715206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b35c3b-9333-4694-b760-073a05807a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.715873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b35c3b-9333-4694-b760-073a05807a2b admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.716153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b35c3b-9333-4694-b760-073a05807a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.716417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b35c3b-9333-4694-b760-073a05807a2b admin admin] PATH_INFO: `/v3/projects/ea470fe7abf94f548ce628919913d02c/users/27b6941e3bb74f47928081fdd42d946c/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.733679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b35c3b-9333-4694-b760-073a05807a2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea470fe7abf94f548ce628919913d02c, user_id=27b6941e3bb74f47928081fdd42d946c, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.736404 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b35c3b-9333-4694-b760-073a05807a2b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.815314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1015/2082] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:21 2022] PUT /identity/v3/projects/ea470fe7abf94f548ce628919913d02c/users/27b6941e3bb74f47928081fdd42d946c/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:21.828418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8684b-aaae-43cb-97eb-f8fb00d649e1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.828897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8684b-aaae-43cb-97eb-f8fb00d649e1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.829243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8684b-aaae-43cb-97eb-f8fb00d649e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.829356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1582e83a-c213-4430-808c-9cbf67b2b7f4 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.829587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1582e83a-c213-4430-808c-9cbf67b2b7f4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.829858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1582e83a-c213-4430-808c-9cbf67b2b7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.850719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e5d8684b-aaae-43cb-97eb-f8fb00d649e1 None None] MFA Rules not processed for user `c6082edc5c234fd1848b58b431c7cf4b`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:21.852053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e5d8684b-aaae-43cb-97eb-f8fb00d649e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.865017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1582e83a-c213-4430-808c-9cbf67b2b7f4 None None] MFA Rules not processed for user `27b6941e3bb74f47928081fdd42d946c`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:21.866239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1582e83a-c213-4430-808c-9cbf67b2b7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.943936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1068/2083] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:21.987070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1016/2084] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:21.987480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8684b-aaae-43cb-97eb-f8fb00d649e1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.046039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:22.065762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8684b-aaae-43cb-97eb-f8fb00d649e1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.070641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.071483 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.073092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.073092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.073092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.074658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.182974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1069/2085] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/projects => generated 383 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:22.186747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1582e83a-c213-4430-808c-9cbf67b2b7f4 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.198644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1582e83a-c213-4430-808c-9cbf67b2b7f4 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.260841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.261552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.261900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.262235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.262971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.265098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.376736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1017/2086] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:22 2022] POST /identity/v3/users => generated 292 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:22.380444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.393060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971e360f-f0dc-4913-a665-204124440fd1 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.398106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.398761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.399082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.399399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.400220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.402120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.421821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1070/2087] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:22 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:22.425936 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.442001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c94391-4cd0-449a-9c4d-01f4d398c464 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.450770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.451786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.452307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.460143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.460143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.460143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.470236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1018/2088] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:22 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:22.474550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.486160 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1776453-780b-4aa4-bac6-ab2f60356932 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.491883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e57d0a5-2616-43f6-b6a6-3e1fede35df5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.493302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e57d0a5-2616-43f6-b6a6-3e1fede35df5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.515846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e57d0a5-2616-43f6-b6a6-3e1fede35df5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.515846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e57d0a5-2616-43f6-b6a6-3e1fede35df5 admin admin] PATH_INFO: `/v3/projects/2c0054e1ee0c41d8aa9a84d25337f4c5/users/b6687b98576e4ab6869346a74818e83b/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.515846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e57d0a5-2616-43f6-b6a6-3e1fede35df5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c0054e1ee0c41d8aa9a84d25337f4c5, user_id=b6687b98576e4ab6869346a74818e83b, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.516759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e57d0a5-2616-43f6-b6a6-3e1fede35df5 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.722700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.722700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:22.736027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f77fe1e-7073-42ee-99e6-2d5a166118eb admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.741087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.741797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.742140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.742630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] PATH_INFO: `/v3/users/d86f06e7c1b64cc3826029b80cd6a360` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.744579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d86f06e7c1b64cc3826029b80cd6a360)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.746603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.749294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:22.878336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1071/2089] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:22 2022] PUT /identity/v3/projects/2c0054e1ee0c41d8aa9a84d25337f4c5/users/b6687b98576e4ab6869346a74818e83b/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:22.878336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47682543-f74b-42f5-ae93-af5008e90046 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.878336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47682543-f74b-42f5-ae93-af5008e90046 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.878336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47682543-f74b-42f5-ae93-af5008e90046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.918589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-47682543-f74b-42f5-ae93-af5008e90046 None None] MFA Rules not processed for user `b6687b98576e4ab6869346a74818e83b`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:22.920230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-47682543-f74b-42f5-ae93-af5008e90046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:23.027388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd86f06e7c1b64cc3826029b80cd6a360'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:23.034312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1072/2090] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:23.041463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47682543-f74b-42f5-ae93-af5008e90046 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.046063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd86f06e7c1b64cc3826029b80cd6a360'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:23.047621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd86f06e7c1b64cc3826029b80cd6a360'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:23.049727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd86f06e7c1b64cc3826029b80cd6a360'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:23.148896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47682543-f74b-42f5-ae93-af5008e90046 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.155103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.155997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.156491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.156974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.158660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.161424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.169029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24f504fc-f47b-40f5-ab1f-73acd7e3d14a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd86f06e7c1b64cc3826029b80cd6a360'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:23.230083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1019/2091] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:22 2022] DELETE /identity/v3/users/d86f06e7c1b64cc3826029b80cd6a360 => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:23.233385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.233885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.234250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.259432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1073/2092] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:23 2022] POST /identity/v3/projects => generated 383 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:23.260949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:23.262445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:23.263656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.265785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:23.282131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:23.369984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff6042-ce31-4f89-ab8a-a3536c615c2d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.375937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.376913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.377510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.378007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.379060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.381438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.415887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1020/2093] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:23.423010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.425012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:23.500835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1def2390-1d98-4b7a-88a0-52b679112a1f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.505040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.506266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.507053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.507576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.519061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.520916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.534694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1021/2094] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:23.549765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1074/2095] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:23 2022] POST /identity/v3/users => generated 298 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:23.551344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.575501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa65980f-11ce-4fff-bd85-3408fc7e33e2 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.588109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.589462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.589792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.590067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.592257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.593952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.607478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1022/2096] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:23 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:23.611685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.630200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6e5330-8dfd-496a-844a-24b9243c644c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.635964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.636922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.637207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.637762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] PATH_INFO: `/v3/projects/f7a91570dfff42419906d529c849eed5/users/600f0684507b4bdc8b4ac4f2821ff745/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.659564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7a91570dfff42419906d529c849eed5, user_id=600f0684507b4bdc8b4ac4f2821ff745, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.662374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.756147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1075/2097] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:23 2022] PUT /identity/v3/projects/f7a91570dfff42419906d529c849eed5/users/600f0684507b4bdc8b4ac4f2821ff745/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:23.761428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.773211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7881ee-d937-4262-92d7-9d3eef8313ff admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.806168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:23.865396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3c71aa-e392-4566-bfdc-800f1e8b0f4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.866301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3c71aa-e392-4566-bfdc-800f1e8b0f4d admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.866605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3c71aa-e392-4566-bfdc-800f1e8b0f4d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.866879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3c71aa-e392-4566-bfdc-800f1e8b0f4d admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.867516 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3c71aa-e392-4566-bfdc-800f1e8b0f4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.870205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3c71aa-e392-4566-bfdc-800f1e8b0f4d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.883231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1023/2098] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:23 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:23.887846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.909620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3631dde1-108a-482e-97fb-5d3c1f8c7b9c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.942322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.942322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.942322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.942322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] PATH_INFO: `/v3/projects/f7a91570dfff42419906d529c849eed5/users/600f0684507b4bdc8b4ac4f2821ff745/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.945977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7a91570dfff42419906d529c849eed5, user_id=600f0684507b4bdc8b4ac4f2821ff745, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.957000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.995115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1076/2099] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:23 2022] PUT /identity/v3/projects/f7a91570dfff42419906d529c849eed5/users/600f0684507b4bdc8b4ac4f2821ff745/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:24.006207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:24.006582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:24.006856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:24.051901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:24.054920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:24.072177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] MFA Rules not processed for user `600f0684507b4bdc8b4ac4f2821ff745`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:24.074527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:24.218514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e503e-a1f6-4269-959d-4278c59adae5 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:24.228734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:24.229675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:24.229952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:24.230221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] PATH_INFO: `/v3/projects/00b0fcbba3a14cde9a5a5cfd0374d3c2` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:24.231501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=00b0fcbba3a14cde9a5a5cfd0374d3c2)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:24.233545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:24.250818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1024/2100] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:24 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:24.269536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:24.271943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:24.338655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc557cf-f499-49d9-9e1d-c1c0411e5947 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:24.343789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ab82fd-558d-4e08-bd48-4f8268e06d7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:24.352762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab82fd-558d-4e08-bd48-4f8268e06d7d service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:24.352762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab82fd-558d-4e08-bd48-4f8268e06d7d service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:24.352762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab82fd-558d-4e08-bd48-4f8268e06d7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:24.352762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-91ab82fd-558d-4e08-bd48-4f8268e06d7d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:24.365577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00b0fcbba3a14cde9a5a5cfd0374d3c2', 'request_id': 'req-cdb17753-b5b9-4155-b449-2229f2008f35'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:24.379031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00b0fcbba3a14cde9a5a5cfd0374d3c2', 'request_id': 'req-cdb17753-b5b9-4155-b449-2229f2008f35'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:24.458267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ab82fd-558d-4e08-bd48-4f8268e06d7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:24.473206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ab82fd-558d-4e08-bd48-4f8268e06d7d service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:24.473651 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00b0fcbba3a14cde9a5a5cfd0374d3c2', 'request_id': 'req-cdb17753-b5b9-4155-b449-2229f2008f35'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:24.478137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=88,keystone:INSERT=6,keystone:DELETE=6 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:24.494021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1025/2101] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:24 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:24.496702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] The token cache is being invalidate because project 00b0fcbba3a14cde9a5a5cfd0374d3c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:24.496702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:24.503009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cdb17753-b5b9-4155-b449-2229f2008f35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:24.508709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1077/2102] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:24 2022] DELETE /identity/v3/projects/00b0fcbba3a14cde9a5a5cfd0374d3c2 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:24.510696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed632c-a5d8-4018-b226-c87b82242712 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:24.511081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed632c-a5d8-4018-b226-c87b82242712 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:24.511453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed632c-a5d8-4018-b226-c87b82242712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:24.545370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b1ed632c-a5d8-4018-b226-c87b82242712 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:24.547071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b1ed632c-a5d8-4018-b226-c87b82242712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:24.687075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1026/2103] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:24.688738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991bc1d-3573-438e-9b48-30c1881e9888 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:24.689028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991bc1d-3573-438e-9b48-30c1881e9888 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:24.689304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991bc1d-3573-438e-9b48-30c1881e9888 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:24.689412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:24.729074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-5991bc1d-3573-438e-9b48-30c1881e9888 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:24.732382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5991bc1d-3573-438e-9b48-30c1881e9888 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:24.748637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1078/2104] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:24.752899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed632c-a5d8-4018-b226-c87b82242712 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:24.775149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed632c-a5d8-4018-b226-c87b82242712 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:24.787231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:24.787860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:24.788139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:24.788406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:24.789065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:24.789361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:24.791703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:24.805459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1027/2105] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:24.813225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5991bc1d-3573-438e-9b48-30c1881e9888 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:24.819059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5991bc1d-3573-438e-9b48-30c1881e9888 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:24.824497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:24.825105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:24.825382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:24.825652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:24.826276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:24.828295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:24.903481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1079/2106] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:24 2022] POST /identity/v3/projects => generated 381 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:24.907594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:24.919984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5cfa95-fe17-43e9-9413-36428a767b89 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:25.041959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:25.047098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:25.052109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:25.052109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:25.052109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:25.052109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:25.134697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1028/2107] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:24 2022] POST /identity/v3/users => generated 291 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:25.140420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:25.173390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462885a2-11ed-4a41-a674-35e54b10d15e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:25.182977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:25.183765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:25.184161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:25.184535 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:25.185276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:25.187200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:25.202073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1080/2108] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:25 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:25.206100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:25.226240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba3b271-5b1f-4fbe-8086-c3ffdaee7001 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:25.235316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fd8ed4-6e54-4e72-87b2-57432bef3e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:25.235940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fd8ed4-6e54-4e72-87b2-57432bef3e77 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:25.236223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fd8ed4-6e54-4e72-87b2-57432bef3e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:25.236496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fd8ed4-6e54-4e72-87b2-57432bef3e77 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:25.237382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fd8ed4-6e54-4e72-87b2-57432bef3e77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:25.239332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fd8ed4-6e54-4e72-87b2-57432bef3e77 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:25.254957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1029/2109] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:25 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:25.259827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:25.296091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbda16dc-2b0c-4895-8437-b69a8b81ec91 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:25.320091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:25.322671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:25.323132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:25.323567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] PATH_INFO: `/v3/projects/b1f2fb36dae5483ebc61c88336461b18/users/2677a4d76de24a729b87a211b49b965a/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:25.345051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1f2fb36dae5483ebc61c88336461b18, user_id=2677a4d76de24a729b87a211b49b965a, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:25.349331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:25.453048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1081/2110] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:25 2022] PUT /identity/v3/projects/b1f2fb36dae5483ebc61c88336461b18/users/2677a4d76de24a729b87a211b49b965a/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:25.456912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:25.457925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:25.459345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:25.503623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:25.503623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:25.581883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:25.596794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1030/2111] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:25.604257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:25.606228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:25.681776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7639ec3d-f095-432e-b695-65cc260ead8b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:25.685389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:25.685992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:25.686341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:25.690380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:25.695284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:25.696749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:25.706005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1082/2112] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:25.730114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:25.731971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:25.745233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ee206f-5a63-45a8-ad38-a197975412f6 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:25.749846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:25.750506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:25.750783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:25.751049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] PATH_INFO: `/v3/users/1dcf55e151b94d379bcf21073f574af5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:25.752170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dcf55e151b94d379bcf21073f574af5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:25.754197 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:25.974557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dcf55e151b94d379bcf21073f574af5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:25.982820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dcf55e151b94d379bcf21073f574af5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:25.983816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dcf55e151b94d379bcf21073f574af5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:25.984529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dcf55e151b94d379bcf21073f574af5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.062768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:26.064193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:26.077762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db4165-0f03-4ec6-ad57-da1647b9fedb service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:26.082357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:26.083017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:26.083300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:26.083580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] PATH_INFO: `/v3/users/8e73f816524a41b0928bded054e5b128` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:26.084693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e73f816524a41b0928bded054e5b128)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:26.086830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:26.122343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-cff5f059-793c-4af8-8b36-e2d04a05967e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dcf55e151b94d379bcf21073f574af5'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.142183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1031/2113] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:25 2022] DELETE /identity/v3/users/1dcf55e151b94d379bcf21073f574af5 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:26.344778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e73f816524a41b0928bded054e5b128'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.357965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e73f816524a41b0928bded054e5b128'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.358748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e73f816524a41b0928bded054e5b128'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.359581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e73f816524a41b0928bded054e5b128'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.485012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e73f816524a41b0928bded054e5b128'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.517545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1083/2114] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:26 2022] DELETE /identity/v3/users/8e73f816524a41b0928bded054e5b128 => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:26.521792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:26.522286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:26.522387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:26.548252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:26.549477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:26.630440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1032/2115] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:26.641147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:26.642744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:26.648495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:26.707305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:26.709059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:26.718426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a664c4-d0b0-4ae2-ba9f-cd04d9e9f27f admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:26.725126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2a9372-2bd5-4cf5-b979-680458d3b752 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:26.725673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:26.727424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:26.727955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:26.728286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:26.743468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:26.750832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:26.751159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:26.751247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:26.751247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:26.751247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] PATH_INFO: `/v3/projects/084b278155af4b28819357bcf190522d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:26.751247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=084b278155af4b28819357bcf190522d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:26.762201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1084/2116] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:26.763023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:26.773500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:26.777861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:26.863509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfb99bc-77d2-4e4b-a03e-f35d9844d5e6 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:26.870633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947837a6-5bc7-412a-a8f8-e415fb7b17f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:26.871757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947837a6-5bc7-412a-a8f8-e415fb7b17f6 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:26.872299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947837a6-5bc7-412a-a8f8-e415fb7b17f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:26.873005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947837a6-5bc7-412a-a8f8-e415fb7b17f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:26.875469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-947837a6-5bc7-412a-a8f8-e415fb7b17f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:26.894177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '084b278155af4b28819357bcf190522d', 'request_id': 'req-30f823e1-10d4-4507-8286-e970b5d920fa'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:26.942439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947837a6-5bc7-412a-a8f8-e415fb7b17f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:26.944131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947837a6-5bc7-412a-a8f8-e415fb7b17f6 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:26.956001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1085/2117] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:27.024220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '084b278155af4b28819357bcf190522d', 'request_id': 'req-30f823e1-10d4-4507-8286-e970b5d920fa'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.036981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '084b278155af4b28819357bcf190522d', 'request_id': 'req-30f823e1-10d4-4507-8286-e970b5d920fa'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.039406 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:27.054248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] The token cache is being invalidate because project 084b278155af4b28819357bcf190522d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:27.054767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.057786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-30f823e1-10d4-4507-8286-e970b5d920fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.060940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1033/2118] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:26 2022] DELETE /identity/v3/projects/084b278155af4b28819357bcf190522d => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:27.070910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f22d506-8970-4fe5-86f1-bd640ddeec50 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.072230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f22d506-8970-4fe5-86f1-bd640ddeec50 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.072230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f22d506-8970-4fe5-86f1-bd640ddeec50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.095999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6f22d506-8970-4fe5-86f1-bd640ddeec50 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:27.097357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6f22d506-8970-4fe5-86f1-bd640ddeec50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.188741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1086/2119] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:27.197018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441ecff8-2f82-4b77-8fc3-ec410944a147 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.197297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441ecff8-2f82-4b77-8fc3-ec410944a147 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.197880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441ecff8-2f82-4b77-8fc3-ec410944a147 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.218899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-441ecff8-2f82-4b77-8fc3-ec410944a147 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:27.220780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-441ecff8-2f82-4b77-8fc3-ec410944a147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.234898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1034/2120] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:27.239455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f22d506-8970-4fe5-86f1-bd640ddeec50 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.250459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f22d506-8970-4fe5-86f1-bd640ddeec50 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.257428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.257428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.257428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.257428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.258059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:27.259053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.260901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.281962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1087/2121] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:53:27.286311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441ecff8-2f82-4b77-8fc3-ec410944a147 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.296101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441ecff8-2f82-4b77-8fc3-ec410944a147 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.301087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.301795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.302270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.302777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.303488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.307630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.312642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.320079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.342982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9d543f-c499-496c-8a6f-81cc48e623f4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.348649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.349350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.349629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.349827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] PATH_INFO: `/v3/projects/4762ef4c48804923bc556537c5ee651c` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.351223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4762ef4c48804923bc556537c5ee651c)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.354341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.436387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1035/2122] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:27 2022] POST /identity/v3/projects => generated 375 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:27.437703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.496551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4762ef4c48804923bc556537c5ee651c', 'request_id': 'req-57cb550d-9660-4867-b1d3-03c715ec9381'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.522935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4762ef4c48804923bc556537c5ee651c', 'request_id': 'req-57cb550d-9660-4867-b1d3-03c715ec9381'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.553315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea1d008-125f-4238-b37d-495ecf227e80 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.589660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4762ef4c48804923bc556537c5ee651c', 'request_id': 'req-57cb550d-9660-4867-b1d3-03c715ec9381'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.614008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] The token cache is being invalidate because project 4762ef4c48804923bc556537c5ee651c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:27.614690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.616747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-57cb550d-9660-4867-b1d3-03c715ec9381 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:27.619969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1088/2123] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:27 2022] DELETE /identity/v3/projects/4762ef4c48804923bc556537c5ee651c => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:27.626491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.627264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.627741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.662722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c745dece-e9c2-4e40-8979-12e1eb5ce6e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.663805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c745dece-e9c2-4e40-8979-12e1eb5ce6e2 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.666259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c745dece-e9c2-4e40-8979-12e1eb5ce6e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.666259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c745dece-e9c2-4e40-8979-12e1eb5ce6e2 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.666259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c745dece-e9c2-4e40-8979-12e1eb5ce6e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.668530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c745dece-e9c2-4e40-8979-12e1eb5ce6e2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.692519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] MFA Rules not processed for user `2677a4d76de24a729b87a211b49b965a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:27.694787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.757795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1036/2124] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:27 2022] POST /identity/v3/users => generated 288 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:27.760330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eb3650-7483-43a0-9cd6-fa20497670ef None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.761154 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eb3650-7483-43a0-9cd6-fa20497670ef None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.761666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eb3650-7483-43a0-9cd6-fa20497670ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.788945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-21eb3650-7483-43a0-9cd6-fa20497670ef None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:27.796008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-21eb3650-7483-43a0-9cd6-fa20497670ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.806251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1089/2125] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:27 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:27.807165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.808484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.815003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1037/2126] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:27.816898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618ce3e-079a-44fa-bcd8-c2d92d7f3bf4 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.817207 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618ce3e-079a-44fa-bcd8-c2d92d7f3bf4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.817446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618ce3e-079a-44fa-bcd8-c2d92d7f3bf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.832534 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6e3602-b4b9-43de-8477-3a6ac013ea94 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.838523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.839406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.839805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.840171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.840930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.842701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.843321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7618ce3e-079a-44fa-bcd8-c2d92d7f3bf4 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:27.844706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7618ce3e-079a-44fa-bcd8-c2d92d7f3bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.857834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1090/2127] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:27 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:27.857834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.858092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1038/2128] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:27.858942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7618ce3e-079a-44fa-bcd8-c2d92d7f3bf4 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.861122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:27.871010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7618ce3e-079a-44fa-bcd8-c2d92d7f3bf4 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.876277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.876891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.877168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.879720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.879720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.881032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.901971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1039/2129] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:27 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:27.901971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.915691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bacc2f-ef53-45cb-ad0b-60ff9f00d5ac admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.923555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.924357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.924634 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.924902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.925546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:27.925843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.933365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.942581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02fad56-f288-41cd-8b5a-a6cbfe220e77 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.947514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1040/2130] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:27.951554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:27.958877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.959676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.959949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.960216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:27.968773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b204b4c-560c-4e39-9763-a167c0d4ee64 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:27.975245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:27.976298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:27.980209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2746021-1b50-4cb3-8681-a2aef410a74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:27.981908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2746021-1b50-4cb3-8681-a2aef410a74c admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:27.982694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2746021-1b50-4cb3-8681-a2aef410a74c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:27.983173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2746021-1b50-4cb3-8681-a2aef410a74c admin admin] PATH_INFO: `/v3/projects/bc8433b42fb74e8bbfef92acedf01259/users/cb93504f2605489eaf50cf10930b3e46/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.001375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1091/2131] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:27 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:28.002353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:28.007280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2746021-1b50-4cb3-8681-a2aef410a74c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc8433b42fb74e8bbfef92acedf01259, user_id=cb93504f2605489eaf50cf10930b3e46, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:28.016113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2746021-1b50-4cb3-8681-a2aef410a74c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:28.018614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e242e-dd36-4a73-9365-14550f7f730d service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:28.023614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:28.025376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.025376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.025577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.029526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:28.029526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:28.113044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1041/2132] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:27 2022] PUT /identity/v3/projects/bc8433b42fb74e8bbfef92acedf01259/users/cb93504f2605489eaf50cf10930b3e46/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:28.118621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.119150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.119493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.140517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:28.141809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:28.174885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1092/2133] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:28 2022] POST /identity/v3/projects => generated 361 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:28.179192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:28.221175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1042/2134] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:28.235367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:28.237312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:28.257742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee251af-065c-4041-9860-d22c4efaed19 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:28.263653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:28.264289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.264750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.265026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.266033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:28.267618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:28.313806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa83361-618a-4eec-9350-480dc3dd6935 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:28.369847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1093/2135] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:28 2022] POST /identity/v3/users => generated 281 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:28.373869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:28.386494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:28.389169 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.389669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.390115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.398945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b4f8e-e81c-4e65-966b-a4981b02f7b6 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:28.409494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:28.410118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.410398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.410698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.411354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:28.412965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:28.414276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:28.416610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:28.424572 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1094/2136] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:28 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:28.431018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:28.431181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1043/2137] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:28.441031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b9c269-7e77-458f-8ab4-9f262ddcf01e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:28.450065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7ece1e-71f6-426d-b2e6-5515289ad8ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:28.451302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7ece1e-71f6-426d-b2e6-5515289ad8ff admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.452161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7ece1e-71f6-426d-b2e6-5515289ad8ff admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.453007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7ece1e-71f6-426d-b2e6-5515289ad8ff admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.453771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7ece1e-71f6-426d-b2e6-5515289ad8ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:28.455556 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7ece1e-71f6-426d-b2e6-5515289ad8ff admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:28.474682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1095/2138] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:28 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:28.477707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:28.489301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd8ca03-4220-459f-a7b4-efa089e5002b service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:28.495434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:28.496138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.496419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.496692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] PATH_INFO: `/v3/projects/41670ad7c88447a5a06e8d64a63c369c/users/789723d780bb43958c29a5392385291b/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.512760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=41670ad7c88447a5a06e8d64a63c369c, user_id=789723d780bb43958c29a5392385291b, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:28.515505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:28.653740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1044/2139] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:28 2022] PUT /identity/v3/projects/41670ad7c88447a5a06e8d64a63c369c/users/789723d780bb43958c29a5392385291b/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:28.665770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286eb819-2600-4fe1-82b9-712b5c538b3a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.666222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286eb819-2600-4fe1-82b9-712b5c538b3a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.666647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286eb819-2600-4fe1-82b9-712b5c538b3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.685666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:28.719526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-286eb819-2600-4fe1-82b9-712b5c538b3a None None] MFA Rules not processed for user `789723d780bb43958c29a5392385291b`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:28.721516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-286eb819-2600-4fe1-82b9-712b5c538b3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:28.839347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1096/2140] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:28 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:28.847201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:28.849239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:28.971004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbd9d39-ac2e-42b8-8607-6d02a0cb67a5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:28.976652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:28.977416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:28.977779 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:28.978170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:28.980051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:29.019807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=1,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:29.027083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286eb819-2600-4fe1-82b9-712b5c538b3a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:29.040233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286eb819-2600-4fe1-82b9-712b5c538b3a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:29.049698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ed3be1-3d82-4670-b61b-6dfe72adf74b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:29.050711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed3be1-3d82-4670-b61b-6dfe72adf74b service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:29.051416 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed3be1-3d82-4670-b61b-6dfe72adf74b service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:29.051866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed3be1-3d82-4670-b61b-6dfe72adf74b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:29.057801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:29.059363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:29.064370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ed3be1-3d82-4670-b61b-6dfe72adf74b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:29.065796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ed3be1-3d82-4670-b61b-6dfe72adf74b service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:29.072915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1045/2141] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:28 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:29.077048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1097/2142] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:29 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:29.096885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:29.177711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9a89bd-f14c-4df9-acc3-2357d6e7dc40 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:29.185396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:29.187098 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:29.187891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:29.188348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] PATH_INFO: `/v3/users/789723d780bb43958c29a5392385291b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:29.189894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] RBAC: Authorizing `identity:delete_user(user_id=789723d780bb43958c29a5392385291b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:29.192837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:29.407042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:29.407842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '789723d780bb43958c29a5392385291b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:29.423262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '789723d780bb43958c29a5392385291b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:29.424918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '789723d780bb43958c29a5392385291b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:29.424918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '789723d780bb43958c29a5392385291b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:29.538044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '789723d780bb43958c29a5392385291b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:29.562344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1046/2143] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:29 2022] DELETE /identity/v3/users/789723d780bb43958c29a5392385291b => generated 0 bytes in 466 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:29.567218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c60bfe-70c3-46ac-9c64-79f496bd6501 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:29.567727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c60bfe-70c3-46ac-9c64-79f496bd6501 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:29.568288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c60bfe-70c3-46ac-9c64-79f496bd6501 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:29.594278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a7c60bfe-70c3-46ac-9c64-79f496bd6501 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:29.595764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a7c60bfe-70c3-46ac-9c64-79f496bd6501 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:29.666149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1098/2144] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:29.700249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:29.701808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:29.775461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57071dac-6395-4100-a008-3da1abd40dcd admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:29.779069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:29.779691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:29.779972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:29.780239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:29.789416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:29.791450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:29.801257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1047/2145] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:30.087780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858eb475-229a-4f77-bf76-ed4364ed7769 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:30.090835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858eb475-229a-4f77-bf76-ed4364ed7769 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:30.090835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858eb475-229a-4f77-bf76-ed4364ed7769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:30.110130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-858eb475-229a-4f77-bf76-ed4364ed7769 None None] MFA Rules not processed for user `2677a4d76de24a729b87a211b49b965a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:30.111401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-858eb475-229a-4f77-bf76-ed4364ed7769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:30.211019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1099/2146] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:30 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:30.491819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:30.494740 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:30.517220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d93963-c273-410e-9f2a-6b0416237629 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:30.526565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:30.527898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:30.528344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:30.528928 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] PATH_INFO: `/v3/projects/41670ad7c88447a5a06e8d64a63c369c` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:30.530583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=41670ad7c88447a5a06e8d64a63c369c)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:30.533898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:30.643159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41670ad7c88447a5a06e8d64a63c369c', 'request_id': 'req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:30.743604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41670ad7c88447a5a06e8d64a63c369c', 'request_id': 'req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:30.754680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41670ad7c88447a5a06e8d64a63c369c', 'request_id': 'req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:30.787246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] The token cache is being invalidate because project 41670ad7c88447a5a06e8d64a63c369c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:30.787930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:30.788658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7e4a0d5c-e100-42db-adbe-fff0fad9efb9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:30.791413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1048/2147] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:30 2022] DELETE /identity/v3/projects/41670ad7c88447a5a06e8d64a63c369c => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:30.796415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a66250-bcdc-4caa-87b9-652d011843f6 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:30.796894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a66250-bcdc-4caa-87b9-652d011843f6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:30.796984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a66250-bcdc-4caa-87b9-652d011843f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:30.827598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d1a66250-bcdc-4caa-87b9-652d011843f6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:30.829424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d1a66250-bcdc-4caa-87b9-652d011843f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:30.940508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1100/2148] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:30.942917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb62c4-3ee1-4ae2-8037-76cd51e2532f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:30.943338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb62c4-3ee1-4ae2-8037-76cd51e2532f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:30.943938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb62c4-3ee1-4ae2-8037-76cd51e2532f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:30.972068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-3ceb62c4-3ee1-4ae2-8037-76cd51e2532f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:30.973420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3ceb62c4-3ee1-4ae2-8037-76cd51e2532f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:30.982846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1049/2149] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:30.989023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a66250-bcdc-4caa-87b9-652d011843f6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:30.999160 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a66250-bcdc-4caa-87b9-652d011843f6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:31.005995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:31.010949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.011238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.011514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.012353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:31.012728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:31.014462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:31.027188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1101/2150] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:31.031655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceb62c4-3ee1-4ae2-8037-76cd51e2532f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:31.041774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceb62c4-3ee1-4ae2-8037-76cd51e2532f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:31.046714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:31.047322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.047631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.047926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.048557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:31.050474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:31.181691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1050/2151] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:31 2022] POST /identity/v3/projects => generated 363 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:31.186080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:31.203852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc592ef5-bcbe-410c-a022-107eb96c37fd admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:31.292513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:31.293061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.293337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.293650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.295452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:31.297784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:31.423032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1102/2152] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:31 2022] POST /identity/v3/users => generated 282 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:31.434987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:31.453763 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c377778-4004-4395-92f8-8fcbdff127e8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:31.458907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:31.459576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.459952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.463135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.463575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:31.465299 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:31.484028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:31.484438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1051/2153] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:31.496019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dab59da-16a6-41f1-8a9f-5a5f238d5d05 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:31.501663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fe4c64-8198-4757-bb55-c936ccfa270c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:31.503073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fe4c64-8198-4757-bb55-c936ccfa270c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.503660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fe4c64-8198-4757-bb55-c936ccfa270c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.508917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fe4c64-8198-4757-bb55-c936ccfa270c admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.508917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fe4c64-8198-4757-bb55-c936ccfa270c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:31.508917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fe4c64-8198-4757-bb55-c936ccfa270c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:31.525364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1103/2154] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:31.531795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:31.559649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3ac1c7-5c0f-4f30-ad51-827c260762e9 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:31.570151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:31.571455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.571733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.572045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] PATH_INFO: `/v3/projects/44559d08c464407f91489c220cf2bc81/users/9bef8e86d23a4b57bafba80926f90975/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.580204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:31.596515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] RBAC: Authorizing `identity:create_grant(project_id=44559d08c464407f91489c220cf2bc81, user_id=9bef8e86d23a4b57bafba80926f90975, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:31.599506 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:31.714053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1052/2155] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:31 2022] PUT /identity/v3/projects/44559d08c464407f91489c220cf2bc81/users/9bef8e86d23a4b57bafba80926f90975/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:31.724420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.724705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.724976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.757287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] MFA Rules not processed for user `9bef8e86d23a4b57bafba80926f90975`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:31.759139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:31.814961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:31.834354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1104/2156] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:31 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:31.846848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:31.848603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:31.926576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da32750f-fe19-447c-bead-78e744fbe14b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:31.938354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2959eff0-4f20-4e86-9569-051f34b5f417 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:31.939130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2959eff0-4f20-4e86-9569-051f34b5f417 service glance] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:31.939514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2959eff0-4f20-4e86-9569-051f34b5f417 service glance] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:31.939882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2959eff0-4f20-4e86-9569-051f34b5f417 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:31.956477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2959eff0-4f20-4e86-9569-051f34b5f417 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:31.958327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2959eff0-4f20-4e86-9569-051f34b5f417 service glance] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:31.983767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1053/2157] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:31 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:32.847164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:33.075093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:34.301035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:34.305026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:34.425673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b7f3a0-dcb3-480d-bd1a-c5cbccfd9cb7 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:34.433775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:34.434478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:34.434766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:34.435042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] PATH_INFO: `/v3/users/27b6941e3bb74f47928081fdd42d946c` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:34.437163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=27b6941e3bb74f47928081fdd42d946c)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:34.439387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:34.680113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27b6941e3bb74f47928081fdd42d946c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:34.703507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27b6941e3bb74f47928081fdd42d946c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:34.704273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27b6941e3bb74f47928081fdd42d946c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:34.705255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27b6941e3bb74f47928081fdd42d946c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:34.707454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:34.876227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27b6941e3bb74f47928081fdd42d946c'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:34.935994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1105/2158] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:34 2022] DELETE /identity/v3/users/27b6941e3bb74f47928081fdd42d946c => generated 0 bytes in 632 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:34.936956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:34.937407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:34.937536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:34.980447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:34.982791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:35.058790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1054/2159] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:35.069556 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:35.073771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:35.076593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:35.220627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91098fa9-36bd-422d-bb81-5c809b7f4d0b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:35.236511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7d8c63-ded7-469a-9333-f059c1c407de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:35.237151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7d8c63-ded7-469a-9333-f059c1c407de service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:35.237433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7d8c63-ded7-469a-9333-f059c1c407de service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:35.237741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7d8c63-ded7-469a-9333-f059c1c407de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:35.249545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7d8c63-ded7-469a-9333-f059c1c407de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:35.251056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7d8c63-ded7-469a-9333-f059c1c407de service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:35.276457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1106/2160] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:36.423434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:36.424941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:36.449798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f666c4c-73a5-46f4-9a2e-b149d824be91 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:36.455686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:36.457501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:36.462006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:36.462006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] PATH_INFO: `/v3/projects/ea470fe7abf94f548ce628919913d02c` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:36.462006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea470fe7abf94f548ce628919913d02c)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:36.463844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:36.568763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:36.569473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:36.569473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:36.608240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] MFA Rules not processed for user `cb93504f2605489eaf50cf10930b3e46`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:36.609505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:36.617361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea470fe7abf94f548ce628919913d02c', 'request_id': 'req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:36.740355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea470fe7abf94f548ce628919913d02c', 'request_id': 'req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:36.748397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea470fe7abf94f548ce628919913d02c', 'request_id': 'req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:36.766286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] The token cache is being invalidate because project ea470fe7abf94f548ce628919913d02c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:36.766286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:36.766877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:36.770929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1055/2161] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:36 2022] DELETE /identity/v3/projects/ea470fe7abf94f548ce628919913d02c => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:36.771975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1107/2162] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:36 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:36.785925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:36.788194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:36.791587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:36.796302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:36.916079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152031fc-10b4-4b20-a27e-403d46d2ebba None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:36.923751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:36.924753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:36.925762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:36.928152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:36.931360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:36.941499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f8ff88-e40f-48d2-a930-b32fbdf2a366 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:36.945372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:36.948692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:36.949429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:36.952498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:36.952777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:36.953052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] PATH_INFO: `/v3/users/b6687b98576e4ab6869346a74818e83b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:36.954357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6687b98576e4ab6869346a74818e83b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:36.956380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:36.960528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1108/2163] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:36 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:37.238985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6687b98576e4ab6869346a74818e83b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:37.247000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6687b98576e4ab6869346a74818e83b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:37.247909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6687b98576e4ab6869346a74818e83b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:37.248487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6687b98576e4ab6869346a74818e83b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:37.358564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6687b98576e4ab6869346a74818e83b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:37.409498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1056/2164] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:36 2022] DELETE /identity/v3/users/b6687b98576e4ab6869346a74818e83b => generated 0 bytes in 622 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:37.591959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:INSERT=1,keystone:DELETE=3,keystone:SELECT=96 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:37.994677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:38.010352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:38.021644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f214cc14-20ae-4bd2-b485-02b7a5891a76 service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:38.129031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:38.129850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:38.132943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:38.132943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] PATH_INFO: `/v3/projects/2c0054e1ee0c41d8aa9a84d25337f4c5` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:38.132943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c0054e1ee0c41d8aa9a84d25337f4c5)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:38.134448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:38.278060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c0054e1ee0c41d8aa9a84d25337f4c5', 'request_id': 'req-55d890fb-8e7c-4d2c-833a-c64780a37e2b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.290903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c0054e1ee0c41d8aa9a84d25337f4c5', 'request_id': 'req-55d890fb-8e7c-4d2c-833a-c64780a37e2b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.413828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c0054e1ee0c41d8aa9a84d25337f4c5', 'request_id': 'req-55d890fb-8e7c-4d2c-833a-c64780a37e2b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.440394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] The token cache is being invalidate because project 2c0054e1ee0c41d8aa9a84d25337f4c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:38.440829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.444787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.449136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1109/2165] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:37 2022] DELETE /identity/v3/projects/2c0054e1ee0c41d8aa9a84d25337f4c5 => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:38.454026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:38.455749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:38.476788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a5fa7e-80be-4096-a510-971d1fb7a781 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:38.556644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:38.558030 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:38.558435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:38.558788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] PATH_INFO: `/v3/users/600f0684507b4bdc8b4ac4f2821ff745` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:38.560013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] RBAC: Authorizing `identity:delete_user(user_id=600f0684507b4bdc8b4ac4f2821ff745)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:38.562706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:38.770498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '600f0684507b4bdc8b4ac4f2821ff745'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.780055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '600f0684507b4bdc8b4ac4f2821ff745'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.781073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '600f0684507b4bdc8b4ac4f2821ff745'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.782603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '600f0684507b4bdc8b4ac4f2821ff745'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.847813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c204382e-0dbd-440f-9cc6-7be08d6f102d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '600f0684507b4bdc8b4ac4f2821ff745'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:38.875152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1057/2166] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:38 2022] DELETE /identity/v3/users/600f0684507b4bdc8b4ac4f2821ff745 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:39.706490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:39.708023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:39.746685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d890fb-8e7c-4d2c-833a-c64780a37e2b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:39.898110 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:39.899632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:39.900068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:39.900482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] PATH_INFO: `/v3/projects/f7a91570dfff42419906d529c849eed5` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:39.903255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7a91570dfff42419906d529c849eed5)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:39.907076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:40.008751 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:40.042910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:40.142092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7a91570dfff42419906d529c849eed5', 'request_id': 'req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:40.150320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7a91570dfff42419906d529c849eed5', 'request_id': 'req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:40.252325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7a91570dfff42419906d529c849eed5', 'request_id': 'req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:40.269184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] The token cache is being invalidate because project f7a91570dfff42419906d529c849eed5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:40.269497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:40.270226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-cd72478c-b8fe-41eb-a2e5-132c52c59ab4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:40.272365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1110/2167] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:39 2022] DELETE /identity/v3/projects/f7a91570dfff42419906d529c849eed5 => generated 0 bytes in 566 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:40.278792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e943a4-6aa2-4b8e-b03e-b1b08840ca74 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:40.279298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e943a4-6aa2-4b8e-b03e-b1b08840ca74 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:40.279748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e943a4-6aa2-4b8e-b03e-b1b08840ca74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:40.310391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-45e943a4-6aa2-4b8e-b03e-b1b08840ca74 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:40.312225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-45e943a4-6aa2-4b8e-b03e-b1b08840ca74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:40.471152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1058/2168] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:40.480647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a355dea-38a7-4f8d-9549-087b54fae3ba None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:40.481027 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a355dea-38a7-4f8d-9549-087b54fae3ba None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:40.481255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a355dea-38a7-4f8d-9549-087b54fae3ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:40.514571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-5a355dea-38a7-4f8d-9549-087b54fae3ba None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:40.516124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5a355dea-38a7-4f8d-9549-087b54fae3ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:40.526368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1111/2169] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:40.532694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e943a4-6aa2-4b8e-b03e-b1b08840ca74 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:40.544582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e943a4-6aa2-4b8e-b03e-b1b08840ca74 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:40.553637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:40.555745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:40.556493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:40.556901 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:40.557851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:40.558254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:40.560002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:40.580513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1059/2170] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:40.581920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a355dea-38a7-4f8d-9549-087b54fae3ba None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:40.614278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a355dea-38a7-4f8d-9549-087b54fae3ba None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:40.619813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb25c89d-738a-44c7-ad1c-b6c1ab1e66a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:40.620563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25c89d-738a-44c7-ad1c-b6c1ab1e66a1 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:40.620947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25c89d-738a-44c7-ad1c-b6c1ab1e66a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:40.621325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25c89d-738a-44c7-ad1c-b6c1ab1e66a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:40.622655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb25c89d-738a-44c7-ad1c-b6c1ab1e66a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:40.625610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb25c89d-738a-44c7-ad1c-b6c1ab1e66a1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:40.746940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1112/2171] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:40 2022] POST /identity/v3/projects => generated 373 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:40.753606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:40.772946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f93b99-907f-4761-9f17-2e7e29bcdd1d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:40.910063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:40.911215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:40.911339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:40.911549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:40.912201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:40.915865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:41.048635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c16b956-edd8-483c-a89c-f843a88906d4 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:41.049196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c16b956-edd8-483c-a89c-f843a88906d4 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:41.049567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c16b956-edd8-483c-a89c-f843a88906d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:41.062030 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1060/2172] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:40 2022] POST /identity/v3/users => generated 287 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:41.070349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6c16b956-edd8-483c-a89c-f843a88906d4 None None] MFA Rules not processed for user `cb93504f2605489eaf50cf10930b3e46`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:41.071383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:41.072647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6c16b956-edd8-483c-a89c-f843a88906d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:41.093402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889ca8c-c213-4f2c-b84e-f27c39e703df admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:41.106342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:41.107017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:41.110664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:41.110664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:41.110664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:41.110664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:41.129090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1061/2173] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:41 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:41.133997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:41.156873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe73f98-2cb0-4cde-a505-e5e58847f41a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:41.167129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cd73f7-cb0a-4382-8984-5d35b7d7a8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:41.167914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cd73f7-cb0a-4382-8984-5d35b7d7a8ef admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:41.170838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cd73f7-cb0a-4382-8984-5d35b7d7a8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:41.170838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cd73f7-cb0a-4382-8984-5d35b7d7a8ef admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:41.170838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cd73f7-cb0a-4382-8984-5d35b7d7a8ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:41.171324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cd73f7-cb0a-4382-8984-5d35b7d7a8ef admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:41.178082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1113/2174] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:41.193114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1062/2175] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:41 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:41.194301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c16b956-edd8-483c-a89c-f843a88906d4 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:41.207268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c16b956-edd8-483c-a89c-f843a88906d4 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:41.214920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:41.215902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:41.216401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:41.216878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] PATH_INFO: `/v3/projects/f83ee172d75e42cbae94076397bc4184/users/88695d6d830f4b478af1e4fd5c5226ef/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:41.244630 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f83ee172d75e42cbae94076397bc4184, user_id=88695d6d830f4b478af1e4fd5c5226ef, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:41.247679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:41.341621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1114/2176] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:41 2022] PUT /identity/v3/projects/f83ee172d75e42cbae94076397bc4184/users/88695d6d830f4b478af1e4fd5c5226ef/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:41.353003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385725e9-81fc-4954-91b2-2ab1e4a3cbf5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:41.353273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385725e9-81fc-4954-91b2-2ab1e4a3cbf5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:41.353637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385725e9-81fc-4954-91b2-2ab1e4a3cbf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:41.417517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-385725e9-81fc-4954-91b2-2ab1e4a3cbf5 None None] MFA Rules not processed for user `88695d6d830f4b478af1e4fd5c5226ef`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:41.419377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-385725e9-81fc-4954-91b2-2ab1e4a3cbf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:41.510158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1063/2177] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:41.521879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:41.524275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:41.611465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eeabae8-6086-4ade-96f5-531d99e1675d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:41.618352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1571b1ce-7ee2-4f5e-9309-f869debc7cc0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:41.619591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1571b1ce-7ee2-4f5e-9309-f869debc7cc0 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:41.619867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1571b1ce-7ee2-4f5e-9309-f869debc7cc0 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:41.620141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1571b1ce-7ee2-4f5e-9309-f869debc7cc0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:41.629748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1571b1ce-7ee2-4f5e-9309-f869debc7cc0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:41.631494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1571b1ce-7ee2-4f5e-9309-f869debc7cc0 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:41.645225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1115/2178] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:41 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:41.685669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385725e9-81fc-4954-91b2-2ab1e4a3cbf5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:41.798733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385725e9-81fc-4954-91b2-2ab1e4a3cbf5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:41.810978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:41.812014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:41.812419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:41.812696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] PATH_INFO: `/v3/users/88695d6d830f4b478af1e4fd5c5226ef` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:41.813882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] RBAC: Authorizing `identity:delete_user(user_id=88695d6d830f4b478af1e4fd5c5226ef)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:41.815877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:42.028378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88695d6d830f4b478af1e4fd5c5226ef'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:42.041309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88695d6d830f4b478af1e4fd5c5226ef'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:42.042067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88695d6d830f4b478af1e4fd5c5226ef'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:42.042708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88695d6d830f4b478af1e4fd5c5226ef'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:42.128689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:42.138900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88695d6d830f4b478af1e4fd5c5226ef'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:42.169388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1064/2179] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:41 2022] DELETE /identity/v3/users/88695d6d830f4b478af1e4fd5c5226ef => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:42.174710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:42.175575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:42.176038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:42.193241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:42.200128 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:42.201597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:42.322260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1116/2180] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:42.335361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:42.337068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:42.412040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c0378-fec8-42b2-8367-b115bc30e26a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:42.420152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf08771-b307-42ca-8d0d-8f542c962104 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:42.420815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf08771-b307-42ca-8d0d-8f542c962104 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:42.421089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf08771-b307-42ca-8d0d-8f542c962104 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:42.421353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf08771-b307-42ca-8d0d-8f542c962104 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:42.431923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf08771-b307-42ca-8d0d-8f542c962104 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:42.433522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf08771-b307-42ca-8d0d-8f542c962104 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:42.443656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1065/2181] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:43.035374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:43.037056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:43.065395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8009a77d-8c0c-45de-b9de-9fc1a941eabe None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:43.071573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:43.072329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.072693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.073039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] PATH_INFO: `/v3/projects/f83ee172d75e42cbae94076397bc4184` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.074255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f83ee172d75e42cbae94076397bc4184)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:43.076716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:43.203275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f83ee172d75e42cbae94076397bc4184', 'request_id': 'req-87a39ec4-44a5-4eba-a196-812cfec86496'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:43.209818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f83ee172d75e42cbae94076397bc4184', 'request_id': 'req-87a39ec4-44a5-4eba-a196-812cfec86496'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:43.280107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f83ee172d75e42cbae94076397bc4184', 'request_id': 'req-87a39ec4-44a5-4eba-a196-812cfec86496'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:43.292129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] The token cache is being invalidate because project f83ee172d75e42cbae94076397bc4184 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:43.292423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:43.293099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-87a39ec4-44a5-4eba-a196-812cfec86496 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:43.294658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1117/2182] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:43 2022] DELETE /identity/v3/projects/f83ee172d75e42cbae94076397bc4184 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:43.299679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa8fc71-b546-4255-a5a8-95616d663a2d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.300152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa8fc71-b546-4255-a5a8-95616d663a2d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.300152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa8fc71-b546-4255-a5a8-95616d663a2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.422025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1aa8fc71-b546-4255-a5a8-95616d663a2d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:43.422025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1aa8fc71-b546-4255-a5a8-95616d663a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:43.489148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1066/2183] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:43.495763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3503fb37-13a1-49d0-900a-f58fe9b63c25 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.496252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3503fb37-13a1-49d0-900a-f58fe9b63c25 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.496629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3503fb37-13a1-49d0-900a-f58fe9b63c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.521305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-3503fb37-13a1-49d0-900a-f58fe9b63c25 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:43.522601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3503fb37-13a1-49d0-900a-f58fe9b63c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:43.532350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1118/2184] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:43.536459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa8fc71-b546-4255-a5a8-95616d663a2d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:43.549638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa8fc71-b546-4255-a5a8-95616d663a2d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:43.556200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:43.557692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.557985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.558264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.559087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:43.559417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:43.561260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:43.577840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1067/2185] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:43.582395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3503fb37-13a1-49d0-900a-f58fe9b63c25 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:43.609540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3503fb37-13a1-49d0-900a-f58fe9b63c25 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:43.615428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:43.616472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.616755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.617029 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.618766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:43.620660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:43.712457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1119/2186] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:43 2022] POST /identity/v3/projects => generated 385 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:43.714052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:43.723985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac57a7c-0a8a-4060-a776-795a67f1b1a5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:43.785574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:43.786183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.786487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.787269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.787390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:43.789304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:43.912009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1068/2187] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:43 2022] POST /identity/v3/users => generated 293 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:43.921064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:43.935291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ad359-04b3-4010-8f7c-66976f04480b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:43.945075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:43.946549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:43.947179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.947507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.947779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.948457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:43.950083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:43.961786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1120/2188] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:43 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:43.966250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:43.976333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bead7aec-bc2d-409a-807e-7a9603d465e4 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:43.982001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f68cd30-ea83-40bc-884e-769d09f4072c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:43.983149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68cd30-ea83-40bc-884e-769d09f4072c admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.983149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68cd30-ea83-40bc-884e-769d09f4072c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.983355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68cd30-ea83-40bc-884e-769d09f4072c admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.984007 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f68cd30-ea83-40bc-884e-769d09f4072c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:43.985604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f68cd30-ea83-40bc-884e-769d09f4072c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:43.997721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1069/2189] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:43 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:44.002363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:44.014259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed0521-bb05-4bcd-8805-50a42a9b4854 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:44.019009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:44.020601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:44.021230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:44.021671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] PATH_INFO: `/v3/projects/f6f4de2e638d4746a56a2d2023d2401b/users/21628ccf9dfb4ad1921aaf27712031c0/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:44.043691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6f4de2e638d4746a56a2d2023d2401b, user_id=21628ccf9dfb4ad1921aaf27712031c0, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:44.047528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:44.165458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1121/2190] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:44 2022] PUT /identity/v3/projects/f6f4de2e638d4746a56a2d2023d2401b/users/21628ccf9dfb4ad1921aaf27712031c0/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:44.170662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257951d3-a728-4840-b6fc-8973aa1c4a96 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:44.170946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257951d3-a728-4840-b6fc-8973aa1c4a96 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:44.171232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257951d3-a728-4840-b6fc-8973aa1c4a96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:44.193668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-257951d3-a728-4840-b6fc-8973aa1c4a96 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:44.195010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-257951d3-a728-4840-b6fc-8973aa1c4a96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:44.277269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1070/2191] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:44.288202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:44.290904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:44.328016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:44.361469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a9ec87-25c6-4438-8801-249f532b5888 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:44.365720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:44.366377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:44.366685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:44.366962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:44.379526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:44.381322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:44.398050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1122/2192] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:48.011211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67140613-1874-4152-a4bb-82006bdb2b9b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:48.012598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67140613-1874-4152-a4bb-82006bdb2b9b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:48.013108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67140613-1874-4152-a4bb-82006bdb2b9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:48.036877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:48.070126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-67140613-1874-4152-a4bb-82006bdb2b9b None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:48.072630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-67140613-1874-4152-a4bb-82006bdb2b9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:48.207114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1071/2193] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:53:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:48.265070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:48.265070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:48.371657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc4a225-44cb-4005-8a28-a29218b8818e service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:48.384968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:48.385946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:48.386231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:48.386554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:48.405780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:48.407861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:48.417996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1123/2194] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:53:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:53:49.574946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51128785-ad3d-43cb-aa06-eea3cee9b6c1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:49.575407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51128785-ad3d-43cb-aa06-eea3cee9b6c1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:49.575480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51128785-ad3d-43cb-aa06-eea3cee9b6c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:49.596040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-51128785-ad3d-43cb-aa06-eea3cee9b6c1 None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:49.597557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-51128785-ad3d-43cb-aa06-eea3cee9b6c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:49.607194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1072/2195] 10.222.0.27 () {58 vars in 967 bytes} [Tue Oct 4 08:53:49 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:49.620792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:49.633758 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97013ec-5f21-467e-b158-0225a5b97cd0 service swift] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:49.647688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:49.648340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:49.648813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:49.649274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:49.666325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:49.668720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:49.685155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1124/2196] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:53:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:53:50.531588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4bd413-d157-4f5a-b217-ee09b8b3032c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:50.531588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4bd413-d157-4f5a-b217-ee09b8b3032c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:50.531588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4bd413-d157-4f5a-b217-ee09b8b3032c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:50.558258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ec4bd413-d157-4f5a-b217-ee09b8b3032c None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:50.559135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ec4bd413-d157-4f5a-b217-ee09b8b3032c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:50.568573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1073/2197] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:53:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:50.587056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:50.601019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c11d909-108c-42b2-aa04-e1b23fa54e89 service swift] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:50.631367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:50.632735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:50.633007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:50.634501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:50.661595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:50.663412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:50.690892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1125/2198] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:53:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:53:52.051280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fa972-43f9-458f-a912-87fdb82c354d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:52.052446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fa972-43f9-458f-a912-87fdb82c354d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:52.053193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0fa972-43f9-458f-a912-87fdb82c354d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:52.123468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0c0fa972-43f9-458f-a912-87fdb82c354d None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:52.135252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0c0fa972-43f9-458f-a912-87fdb82c354d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:52.146292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1074/2199] 10.222.0.27 () {58 vars in 967 bytes} [Tue Oct 4 08:53:52 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:52.162113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:52.184280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a0e379-d017-4842-b892-0c18c91306e1 service swift] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:52.190920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:52.191721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:52.192109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:52.192479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:52.208025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:52.211934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:52.229717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1126/2200] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:53:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:53:52.620353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828da1af-583f-4d94-87aa-f3680b075306 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:52.620826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828da1af-583f-4d94-87aa-f3680b075306 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:52.621236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828da1af-583f-4d94-87aa-f3680b075306 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:52.667000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-828da1af-583f-4d94-87aa-f3680b075306 None None] MFA Rules not processed for user `21628ccf9dfb4ad1921aaf27712031c0`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:52.668217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-828da1af-583f-4d94-87aa-f3680b075306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:52.747132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1075/2201] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:52.751746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:52.764121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e2166a-e70c-49db-80cc-4bfc6ce1eafa service swift] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:52.768803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:52.769472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:52.769752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:52.770022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:52.770672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:52.772543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:52.874563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1127/2202] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:52 2022] POST /identity/v3/projects => generated 383 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:52.878484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828da1af-583f-4d94-87aa-f3680b075306 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:52.892704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828da1af-583f-4d94-87aa-f3680b075306 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:52.978352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:52.980186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:52.980762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:52.981317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:52.982250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:52.985722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:53.079174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1076/2203] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:52 2022] POST /identity/v3/users => generated 298 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:53.080156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:53.094625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384d2551-886a-4bf5-8119-74022d9821f7 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:53.101182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:53.102375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:53.102665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:53.102940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:53.103590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:53.105216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:53.115736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1128/2204] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:53.120586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:53.131532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98c4f0-7ee9-4a21-8bca-e24dc30e7d63 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:53.136831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:53.137683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:53.137909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:53.138336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] PATH_INFO: `/v3/projects/73f71a9ac36e40c38b0e1e7306da9e6d/users/e185df1688f245d098b15c9298e4b5a7/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:53.146047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:53.165758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73f71a9ac36e40c38b0e1e7306da9e6d, user_id=e185df1688f245d098b15c9298e4b5a7, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:53.173055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:53.270143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1077/2205] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:53 2022] PUT /identity/v3/projects/73f71a9ac36e40c38b0e1e7306da9e6d/users/e185df1688f245d098b15c9298e4b5a7/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:53.274691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:53.287618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749bcaca-e5f1-4481-a9bb-34ca16d2f2e5 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:53.357835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:53.370697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:53.371308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:53.371584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:53.371854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:53.372551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:53.374161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:53.384897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1129/2206] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:53 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:53.389413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:53.401731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb0b76-4ef5-40ba-bfc7-0027ceba7ff3 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:53.406934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9155eda-bce4-4cbf-ada4-4547c52bcff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:53.407662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9155eda-bce4-4cbf-ada4-4547c52bcff7 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:53.407932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9155eda-bce4-4cbf-ada4-4547c52bcff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:53.408252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9155eda-bce4-4cbf-ada4-4547c52bcff7 admin admin] PATH_INFO: `/v3/projects/73f71a9ac36e40c38b0e1e7306da9e6d/users/e185df1688f245d098b15c9298e4b5a7/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:53.410776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9155eda-bce4-4cbf-ada4-4547c52bcff7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73f71a9ac36e40c38b0e1e7306da9e6d, user_id=e185df1688f245d098b15c9298e4b5a7, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:53.415344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9155eda-bce4-4cbf-ada4-4547c52bcff7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:53.432152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1078/2207] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:53 2022] PUT /identity/v3/projects/73f71a9ac36e40c38b0e1e7306da9e6d/users/e185df1688f245d098b15c9298e4b5a7/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:57.964123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.967189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:58.025968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce544b55-91ae-465e-88cd-b792246b67c3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:58.238033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:58.239876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:58.240531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:58.241007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] PATH_INFO: `/v3/users/c6082edc5c234fd1848b58b431c7cf4b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:58.243386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6082edc5c234fd1848b58b431c7cf4b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:58.246874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:58.534111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6082edc5c234fd1848b58b431c7cf4b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:58.547680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6082edc5c234fd1848b58b431c7cf4b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:58.548937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6082edc5c234fd1848b58b431c7cf4b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:58.549988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6082edc5c234fd1848b58b431c7cf4b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:58.680597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6082edc5c234fd1848b58b431c7cf4b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:58.718274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1130/2208] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:57 2022] DELETE /identity/v3/users/c6082edc5c234fd1848b58b431c7cf4b => generated 0 bytes in 754 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:58.723939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:58.725025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:58.725629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:58.775732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:58.777276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:58.924866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1079/2209] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:58.957217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:58.959096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:59.046956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175cbbd9-f8b4-4bce-b57d-5a67fdfee394 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:59.051461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a606ed82-a30b-45c8-a402-ffcb62d5ec1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:59.052415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a606ed82-a30b-45c8-a402-ffcb62d5ec1b service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:59.052918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a606ed82-a30b-45c8-a402-ffcb62d5ec1b service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:59.053383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a606ed82-a30b-45c8-a402-ffcb62d5ec1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:59.069181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a606ed82-a30b-45c8-a402-ffcb62d5ec1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:59.072625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a606ed82-a30b-45c8-a402-ffcb62d5ec1b service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:59.088134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1131/2210] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:59.439541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:59.441624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:59.459480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d4f623-69c2-4d4b-a746-d870b8c832ab None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:59.470543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:59.471190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:59.471470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:59.471814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] PATH_INFO: `/v3/projects/242e1450de5b4b7b9cc030474995f021` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:59.472959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=242e1450de5b4b7b9cc030474995f021)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:59.474912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:59.639454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '242e1450de5b4b7b9cc030474995f021', 'request_id': 'req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:59.756592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '242e1450de5b4b7b9cc030474995f021', 'request_id': 'req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:59.771675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '242e1450de5b4b7b9cc030474995f021', 'request_id': 'req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:59.786172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] The token cache is being invalidate because project 242e1450de5b4b7b9cc030474995f021 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:59.786551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:59.787759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8a3a77b3-2afb-4ec1-9587-e0b9db216f3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:59.793039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1080/2211] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:59 2022] DELETE /identity/v3/projects/242e1450de5b4b7b9cc030474995f021 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:59.794960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816885ac-0c97-4584-a6e1-e999a4793da2 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:59.795259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816885ac-0c97-4584-a6e1-e999a4793da2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:59.795538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816885ac-0c97-4584-a6e1-e999a4793da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:59.811433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:59.824811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-816885ac-0c97-4584-a6e1-e999a4793da2 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:59.826105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-816885ac-0c97-4584-a6e1-e999a4793da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:59.901655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1132/2212] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:59.911609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df3be2-cade-4738-a20e-3d3fb2c87b08 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:59.912080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df3be2-cade-4738-a20e-3d3fb2c87b08 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:59.912619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df3be2-cade-4738-a20e-3d3fb2c87b08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:59.941118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-19df3be2-cade-4738-a20e-3d3fb2c87b08 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:59.942661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-19df3be2-cade-4738-a20e-3d3fb2c87b08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:59.956009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1081/2213] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:53:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:59.957313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816885ac-0c97-4584-a6e1-e999a4793da2 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:59.973099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816885ac-0c97-4584-a6e1-e999a4793da2 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:59.988260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:59.988980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:59.989334 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:59.990548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:59.996184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:59.996184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:59.996184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:00.019417 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1133/2214] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:00.024725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df3be2-cade-4738-a20e-3d3fb2c87b08 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:00.034447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df3be2-cade-4738-a20e-3d3fb2c87b08 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:00.040285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:00.041278 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:00.041799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:00.042303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:00.043198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:00.045383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:00.203511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1082/2215] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:00 2022] POST /identity/v3/projects => generated 397 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:00.208425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:00.222728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9042359b-b438-4063-a466-24a99aa8273a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:00.419962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:00.419962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:00.419962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:00.419962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:00.420314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:00.422247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:00.588795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1134/2216] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:00 2022] POST /identity/v3/users => generated 299 bytes in 377 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:00.589486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:00.600533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e50c7b-2eec-443c-898d-52ee0dedc772 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:00.606004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:00.606659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:00.607968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:00.607968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:00.607968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:00.609510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:00.624200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1083/2217] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:00.627059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:00.640512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bb3275-d6bb-435f-a41f-7f46f3596047 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:00.646759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfa0f6e-446b-492d-893c-32e9d834bfd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:00.647612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfa0f6e-446b-492d-893c-32e9d834bfd7 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:00.648532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfa0f6e-446b-492d-893c-32e9d834bfd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:00.649860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfa0f6e-446b-492d-893c-32e9d834bfd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:00.653853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfa0f6e-446b-492d-893c-32e9d834bfd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:00.656368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfa0f6e-446b-492d-893c-32e9d834bfd7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:00.672631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1135/2218] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:00 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:00.676792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:00.694943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906fde9c-3495-416f-a329-529a46a55cf5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:00.704503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:00.705770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:00.706149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:00.707665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] PATH_INFO: `/v3/projects/e1a5bab797674be0b9dcf0c41fd0702f/users/ef27d32b56864f83b93265ccbd03ed64/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:00.741621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:00.752589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1a5bab797674be0b9dcf0c41fd0702f, user_id=ef27d32b56864f83b93265ccbd03ed64, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:00.753317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:00.864730 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1084/2219] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:00 2022] PUT /identity/v3/projects/e1a5bab797674be0b9dcf0c41fd0702f/users/ef27d32b56864f83b93265ccbd03ed64/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:00.880401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebea5e5-09a3-40ad-b692-4767b2081c52 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:00.880401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebea5e5-09a3-40ad-b692-4767b2081c52 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:00.880401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebea5e5-09a3-40ad-b692-4767b2081c52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:00.922182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-eebea5e5-09a3-40ad-b692-4767b2081c52 None None] MFA Rules not processed for user `ef27d32b56864f83b93265ccbd03ed64`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:00.925650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-eebea5e5-09a3-40ad-b692-4767b2081c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:01.017801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:01.027440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1136/2220] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:00 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:01.048245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:01.051796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:01.189753 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a20169-984d-464e-bbc1-2ccc791e7f67 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:01.219114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:01.220114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:01.220954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:01.221662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:01.239560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:01.243838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:01.263626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1085/2221] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:01 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:01.600937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d17f91-e56e-403f-9fbb-45c4b9f79e75 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:01.600937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d17f91-e56e-403f-9fbb-45c4b9f79e75 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:01.600937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d17f91-e56e-403f-9fbb-45c4b9f79e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:01.683068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-78d17f91-e56e-403f-9fbb-45c4b9f79e75 None None] MFA Rules not processed for user `e185df1688f245d098b15c9298e4b5a7`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:01.686089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-78d17f91-e56e-403f-9fbb-45c4b9f79e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:01.813157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1137/2222] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:01 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:01.833524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:01.835328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:01.921268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a9dae7-d09a-4019-88bf-8d677749dc0f service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:01.930528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:01.931442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:01.931972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:01.932460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:01.934453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:02.046066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:02.048062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:02.063125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1086/2223] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:02.458783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d234dd-dd35-4691-b993-446b03421d0e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.459046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d234dd-dd35-4691-b993-446b03421d0e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.459323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d234dd-dd35-4691-b993-446b03421d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.482253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c9d234dd-dd35-4691-b993-446b03421d0e None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:02.483930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c9d234dd-dd35-4691-b993-446b03421d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:02.602846 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1138/2224] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:54:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:02.643530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:02.645955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:02.748908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:02.787883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008638f5-a200-4440-aa42-3cf605ad9907 service glance] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:02.800445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc35aa4-b1d9-45e5-968c-9d6e21ffdf60 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:02.801077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc35aa4-b1d9-45e5-968c-9d6e21ffdf60 service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.801357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc35aa4-b1d9-45e5-968c-9d6e21ffdf60 service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.801628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc35aa4-b1d9-45e5-968c-9d6e21ffdf60 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.812920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc35aa4-b1d9-45e5-968c-9d6e21ffdf60 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:02.814871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc35aa4-b1d9-45e5-968c-9d6e21ffdf60 service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:02.834616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1087/2225] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:54:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:54:12.568505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=59 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:12.902725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=11 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:14.028936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d234dd-dd35-4691-b993-446b03421d0e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.114917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d234dd-dd35-4691-b993-446b03421d0e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.121456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.121456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.121877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.121965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] PATH_INFO: `/v3/users/ef27d32b56864f83b93265ccbd03ed64` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.123254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef27d32b56864f83b93265ccbd03ed64)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.125207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.342072 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef27d32b56864f83b93265ccbd03ed64'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.350380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef27d32b56864f83b93265ccbd03ed64'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.351835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef27d32b56864f83b93265ccbd03ed64'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.352823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef27d32b56864f83b93265ccbd03ed64'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.442475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef27d32b56864f83b93265ccbd03ed64'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.513983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1139/2226] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:14 2022] DELETE /identity/v3/users/ef27d32b56864f83b93265ccbd03ed64 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:14.518894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.519487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.523418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.565976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:14.567714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.676379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1088/2227] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:14.681870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.684581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.751286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301886e-5a80-44d4-883a-5fd72df039a0 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.759506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.759506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.766179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.766179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.772262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.774600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.786008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1140/2228] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:15.530210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.532329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.553488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a73f7a-b027-4ca3-b952-d114920414e3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.561002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.561842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.562125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.562391 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] PATH_INFO: `/v3/projects/e1a5bab797674be0b9dcf0c41fd0702f` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.563738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1a5bab797674be0b9dcf0c41fd0702f)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.565809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.683382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1a5bab797674be0b9dcf0c41fd0702f', 'request_id': 'req-62cef074-0ae0-475c-ae8d-39b28cc39d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.844307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.845771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.862869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85597684-75d9-474b-8646-72f01a3fa10f service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.881834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1a5bab797674be0b9dcf0c41fd0702f', 'request_id': 'req-62cef074-0ae0-475c-ae8d-39b28cc39d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.883532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:15.890010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1a5bab797674be0b9dcf0c41fd0702f', 'request_id': 'req-62cef074-0ae0-475c-ae8d-39b28cc39d6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.902159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] The token cache is being invalidate because project e1a5bab797674be0b9dcf0c41fd0702f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:15.902543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.903255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-62cef074-0ae0-475c-ae8d-39b28cc39d6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.905442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1089/2229] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:15 2022] DELETE /identity/v3/projects/e1a5bab797674be0b9dcf0c41fd0702f => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:15.938217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.938903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.940514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.940514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] PATH_INFO: `/v3/users/9bef8e86d23a4b57bafba80926f90975` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.941769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bef8e86d23a4b57bafba80926f90975)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.943835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.159050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bef8e86d23a4b57bafba80926f90975'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:16.167916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bef8e86d23a4b57bafba80926f90975'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:16.168682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bef8e86d23a4b57bafba80926f90975'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:16.170614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bef8e86d23a4b57bafba80926f90975'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:16.262835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bef8e86d23a4b57bafba80926f90975'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:16.290179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1141/2230] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:15 2022] DELETE /identity/v3/users/9bef8e86d23a4b57bafba80926f90975 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:16.299557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c81a36-e075-4eed-90aa-cb85e0d7cfae None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.299840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c81a36-e075-4eed-90aa-cb85e0d7cfae None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.300113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c81a36-e075-4eed-90aa-cb85e0d7cfae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.321249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-97c81a36-e075-4eed-90aa-cb85e0d7cfae None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:16.322555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-97c81a36-e075-4eed-90aa-cb85e0d7cfae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.387869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1090/2231] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:16.395989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.398523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.398637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a299213-94fe-4b9b-bba5-9825427e216c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.398976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a299213-94fe-4b9b-bba5-9825427e216c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.399384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a299213-94fe-4b9b-bba5-9825427e216c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.427908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9a299213-94fe-4b9b-bba5-9825427e216c None None] MFA Rules not processed for user `21628ccf9dfb4ad1921aaf27712031c0`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:16.429802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9a299213-94fe-4b9b-bba5-9825427e216c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.448987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:16.458712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae994f9-8ef6-405c-900a-d2805475aed6 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.462583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.463196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.463472 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.463743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.472955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.474623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.484266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1142/2232] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:16.492383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1091/2233] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:16 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:17.135138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.136677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.153484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab74c660-b043-43ba-b43e-2e6cb273099f service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.159997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.159997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.159997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.159997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] PATH_INFO: `/v3/projects/44559d08c464407f91489c220cf2bc81` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.161533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44559d08c464407f91489c220cf2bc81)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.163798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.280362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:17.287759 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44559d08c464407f91489c220cf2bc81', 'request_id': 'req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.298573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44559d08c464407f91489c220cf2bc81', 'request_id': 'req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.496430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44559d08c464407f91489c220cf2bc81', 'request_id': 'req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.517941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] The token cache is being invalidate because project 44559d08c464407f91489c220cf2bc81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:17.518228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.519079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9127a026-9c3c-4eb4-a84d-45e0f6a722c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.520813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1143/2234] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:17 2022] DELETE /identity/v3/projects/44559d08c464407f91489c220cf2bc81 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:17.530646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a342c8f0-f1b3-415f-a682-125aa074b1b5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.530646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a342c8f0-f1b3-415f-a682-125aa074b1b5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.530646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a342c8f0-f1b3-415f-a682-125aa074b1b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.546881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a342c8f0-f1b3-415f-a682-125aa074b1b5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:17.548087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a342c8f0-f1b3-415f-a682-125aa074b1b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.630132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1092/2235] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:17.638974 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77fe72b-f495-4825-aacb-07097d5cbb35 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.639324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77fe72b-f495-4825-aacb-07097d5cbb35 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.639604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77fe72b-f495-4825-aacb-07097d5cbb35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.660866 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e77fe72b-f495-4825-aacb-07097d5cbb35 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:17.662189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e77fe72b-f495-4825-aacb-07097d5cbb35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.676022 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1144/2236] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:17.677481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a342c8f0-f1b3-415f-a682-125aa074b1b5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.688864 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a342c8f0-f1b3-415f-a682-125aa074b1b5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.693535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.694132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.700430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.700430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.700430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:17.700430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.700430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.710241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1093/2237] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:17.714895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77fe72b-f495-4825-aacb-07097d5cbb35 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.724085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77fe72b-f495-4825-aacb-07097d5cbb35 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.737833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.737833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.737833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.737833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.738336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.739530 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.867760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1145/2238] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:17 2022] POST /identity/v3/projects => generated 375 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:17.872174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.883151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee2b32-700a-4efb-a8d7-3805cdecdd74 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.941350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.941963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.943171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.943171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.943171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.945095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.032823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1094/2239] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:17 2022] POST /identity/v3/users => generated 288 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:18.035547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.048220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd4f6e1-57f9-489b-b9b5-3ed1edc97393 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.054737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.055357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.055638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.055910 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.056554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.058201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.069161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1146/2240] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:18.074328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.091480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b65bd2b-357b-474b-808c-aa7ec5fba473 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.096352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11424d99-f882-4ddc-ab9a-965a7d64dae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.096998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11424d99-f882-4ddc-ab9a-965a7d64dae5 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.097282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11424d99-f882-4ddc-ab9a-965a7d64dae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.097553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11424d99-f882-4ddc-ab9a-965a7d64dae5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.098224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11424d99-f882-4ddc-ab9a-965a7d64dae5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.099847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11424d99-f882-4ddc-ab9a-965a7d64dae5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.111913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1095/2241] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:18 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:18.116151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.126517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940beedc-f3a0-44e9-9bf0-43393d68b6f3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.132671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.133350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.133657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.133931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] PATH_INFO: `/v3/projects/f696021154794e65bf3cdba31c6f427f/users/c4acc89772f848359873b21649ee1de3/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.157078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f696021154794e65bf3cdba31c6f427f, user_id=c4acc89772f848359873b21649ee1de3, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.159835 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.243282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1147/2242] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:18 2022] PUT /identity/v3/projects/f696021154794e65bf3cdba31c6f427f/users/c4acc89772f848359873b21649ee1de3/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:18.258620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0e533e-0ae2-4b33-9b28-80cc0185577f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.258620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0e533e-0ae2-4b33-9b28-80cc0185577f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.258620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0e533e-0ae2-4b33-9b28-80cc0185577f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.290755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-4f0e533e-0ae2-4b33-9b28-80cc0185577f None None] MFA Rules not processed for user `c4acc89772f848359873b21649ee1de3`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:18.292552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4f0e533e-0ae2-4b33-9b28-80cc0185577f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.321235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:18.363659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1096/2243] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:18.375105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.377081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.442115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b55ac5-985a-4396-9050-bf85aece880e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.446453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.447057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.447785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.447785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.456824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.458365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.473452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1148/2244] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:21.090714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2717b-8302-4c3a-8b27-6c77c7b0befc None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.091187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2717b-8302-4c3a-8b27-6c77c7b0befc None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.091187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2717b-8302-4c3a-8b27-6c77c7b0befc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.124903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-85a2717b-8302-4c3a-8b27-6c77c7b0befc None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:21.126224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-85a2717b-8302-4c3a-8b27-6c77c7b0befc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.207618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1097/2245] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:54:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:21.238453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:21.239917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.327454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007e9f8f-0355-4a4b-8c82-57e1c9bb3910 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:21.332536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:21.333157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.333439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.333709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.341725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:21.358513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:21.362719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:21.375540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1149/2246] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:54:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:54:21.433165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b299357-e734-444e-a99e-75629ea36287 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.433478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b299357-e734-444e-a99e-75629ea36287 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.433728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b299357-e734-444e-a99e-75629ea36287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.455360 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0b299357-e734-444e-a99e-75629ea36287 None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:21.456628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0b299357-e734-444e-a99e-75629ea36287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.465323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1098/2247] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:54:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:21.481600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:21.494189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f749e8c-062b-47f2-b277-4aa9e164fb64 service swift] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:21.500340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e8c461-0861-4695-9b14-f2ecbd536985 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:21.501109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8c461-0861-4695-9b14-f2ecbd536985 service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.501483 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8c461-0861-4695-9b14-f2ecbd536985 service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.501921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8c461-0861-4695-9b14-f2ecbd536985 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.513170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e8c461-0861-4695-9b14-f2ecbd536985 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:21.515150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e8c461-0861-4695-9b14-f2ecbd536985 service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:21.525979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1150/2248] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:54:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:54:23.182006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25678af4-a4ba-46a6-bc12-652946b4dbb9 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.188748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25678af4-a4ba-46a6-bc12-652946b4dbb9 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.188748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25678af4-a4ba-46a6-bc12-652946b4dbb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.216605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-25678af4-a4ba-46a6-bc12-652946b4dbb9 None None] MFA Rules not processed for user `185e359e571f44bda1585124529b36a8`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:23.218004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-25678af4-a4ba-46a6-bc12-652946b4dbb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:23.350356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1099/2249] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:23 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:26.518655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07d9ff3-cd3e-42c3-9a6d-6f6aac34871b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.519161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07d9ff3-cd3e-42c3-9a6d-6f6aac34871b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.519161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07d9ff3-cd3e-42c3-9a6d-6f6aac34871b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.555681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-b07d9ff3-cd3e-42c3-9a6d-6f6aac34871b None None] MFA Rules not processed for user `14d56042415540a3b3ed06578b712ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:26.556941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b07d9ff3-cd3e-42c3-9a6d-6f6aac34871b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:26.696671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1151/2250] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:33.265309 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25678af4-a4ba-46a6-bc12-652946b4dbb9 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.361132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25678af4-a4ba-46a6-bc12-652946b4dbb9 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.366875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.367528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.367807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.368192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] PATH_INFO: `/v3/users/c4acc89772f848359873b21649ee1de3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.369487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4acc89772f848359873b21649ee1de3)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.371451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.388859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:33.602885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4acc89772f848359873b21649ee1de3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:33.613371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4acc89772f848359873b21649ee1de3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:33.614160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4acc89772f848359873b21649ee1de3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:33.614998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4acc89772f848359873b21649ee1de3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:33.698715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4acc89772f848359873b21649ee1de3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:33.720839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1100/2251] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:33 2022] DELETE /identity/v3/users/c4acc89772f848359873b21649ee1de3 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:33.726367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.726985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.726985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.757151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:33.759037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:33.827515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1152/2252] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:33.839418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.841158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:33.918064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995d73a3-38b1-4b23-9329-2ce8910b589e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.922245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb75099e-88bd-4594-a1da-8192085b126c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.923087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75099e-88bd-4594-a1da-8192085b126c service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.923536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75099e-88bd-4594-a1da-8192085b126c service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.923970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb75099e-88bd-4594-a1da-8192085b126c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.938626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb75099e-88bd-4594-a1da-8192085b126c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.940053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb75099e-88bd-4594-a1da-8192085b126c service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.950312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1101/2253] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:34.457302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:34.457302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:34.468829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635a6cb2-e475-4158-bd67-4052ca2e96b2 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:34.474095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:34.475024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:34.475308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:34.475583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] PATH_INFO: `/v3/projects/f696021154794e65bf3cdba31c6f427f` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:34.476922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=f696021154794e65bf3cdba31c6f427f)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:34.479215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:34.629470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f696021154794e65bf3cdba31c6f427f', 'request_id': 'req-4f346e18-858d-4219-80f9-34175e8bc6ea'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.644428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f696021154794e65bf3cdba31c6f427f', 'request_id': 'req-4f346e18-858d-4219-80f9-34175e8bc6ea'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.727666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f696021154794e65bf3cdba31c6f427f', 'request_id': 'req-4f346e18-858d-4219-80f9-34175e8bc6ea'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.741681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] The token cache is being invalidate because project f696021154794e65bf3cdba31c6f427f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:34.742133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.742956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4f346e18-858d-4219-80f9-34175e8bc6ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.744590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1153/2254] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:34 2022] DELETE /identity/v3/projects/f696021154794e65bf3cdba31c6f427f => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:43.945822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=10,keystone:SELECT=48,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:44.737292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=87,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:45.646375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42b180-bb40-49bb-87cd-4fe2c137e505 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:45.646907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42b180-bb40-49bb-87cd-4fe2c137e505 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:45.647011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42b180-bb40-49bb-87cd-4fe2c137e505 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:45.648757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1102/2255] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:54:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:54:45.654727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afd913-e50a-4eb5-a710-12a6337bc862 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:45.655036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afd913-e50a-4eb5-a710-12a6337bc862 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:45.655307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afd913-e50a-4eb5-a710-12a6337bc862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:45.675967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a3afd913-e50a-4eb5-a710-12a6337bc862 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:45.677358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a3afd913-e50a-4eb5-a710-12a6337bc862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:45.746849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1154/2256] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:54:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:45.759189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e42b180-bb40-49bb-87cd-4fe2c137e505 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:45.760942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0e42b180-bb40-49bb-87cd-4fe2c137e505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:45.818490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e42b180-bb40-49bb-87cd-4fe2c137e505 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:45.822292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954beb0a-d0f3-4e6b-ab44-d9ea5360d561 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:45.822927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954beb0a-d0f3-4e6b-ab44-d9ea5360d561 service glance] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:45.823206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954beb0a-d0f3-4e6b-ab44-d9ea5360d561 service glance] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:45.823477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954beb0a-d0f3-4e6b-ab44-d9ea5360d561 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:45.833997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954beb0a-d0f3-4e6b-ab44-d9ea5360d561 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:45.836117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954beb0a-d0f3-4e6b-ab44-d9ea5360d561 service glance] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:45.847106 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1103/2257] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:46.060236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64ad6e4-d7ad-4deb-9ff0-7663f6bb9b1f None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:46.060520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64ad6e4-d7ad-4deb-9ff0-7663f6bb9b1f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:46.060826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64ad6e4-d7ad-4deb-9ff0-7663f6bb9b1f None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:46.062142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1155/2258] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:54:46.068125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed95fc-94f9-4e63-9216-953f4ac8e7c4 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:46.068706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed95fc-94f9-4e63-9216-953f4ac8e7c4 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:46.069206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed95fc-94f9-4e63-9216-953f4ac8e7c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:46.090197 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-feed95fc-94f9-4e63-9216-953f4ac8e7c4 None None] MFA Rules not processed for user `df391d2bf7884538823f04db5564d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:46.091648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-feed95fc-94f9-4e63-9216-953f4ac8e7c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:46.146626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1104/2259] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:54:46 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:46.159314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64ad6e4-d7ad-4deb-9ff0-7663f6bb9b1f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:46.161068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c64ad6e4-d7ad-4deb-9ff0-7663f6bb9b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:46.176783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64ad6e4-d7ad-4deb-9ff0-7663f6bb9b1f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:46.182888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7505d8d9-af37-47dc-82be-d95c5dd01a69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:46.183671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7505d8d9-af37-47dc-82be-d95c5dd01a69 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:46.184000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7505d8d9-af37-47dc-82be-d95c5dd01a69 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:46.184703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7505d8d9-af37-47dc-82be-d95c5dd01a69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:46.196747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7505d8d9-af37-47dc-82be-d95c5dd01a69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:46.196747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7505d8d9-af37-47dc-82be-d95c5dd01a69 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:46.203317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1156/2260] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:46.292935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d32c9b-7ba6-4a4d-8a03-53f730ebe96d None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:46.293333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d32c9b-7ba6-4a4d-8a03-53f730ebe96d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:46.293673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d32c9b-7ba6-4a4d-8a03-53f730ebe96d None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:46.295171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1105/2261] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:54:46.301034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c657a-4149-4330-be99-d506cee2f289 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:46.301315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c657a-4149-4330-be99-d506cee2f289 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:46.305595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c657a-4149-4330-be99-d506cee2f289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:46.322098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-588c657a-4149-4330-be99-d506cee2f289 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:46.323368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-588c657a-4149-4330-be99-d506cee2f289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:46.333356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1157/2262] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:54:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:46.346666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d32c9b-7ba6-4a4d-8a03-53f730ebe96d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:46.358438 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d32c9b-7ba6-4a4d-8a03-53f730ebe96d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:46.361916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:46.362571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:46.362844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:46.363257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:46.374630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:46.376062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:46.387666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1106/2263] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:56.139436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:56.139911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:56.140484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:56.164244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:56.165587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:56.222016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1158/2264] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:54:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:56.247983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:56.251239 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:56.320690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a705f4b1-c6e0-47e6-862d-aa8bfc9c79e3 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:56.326795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:56.327408 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:56.327688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:56.327958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:56.337049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:56.338755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:56.347522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1107/2265] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:54:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:55:06.222368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=65 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:06.344490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=87 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:10.297028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:10.298640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:10.379013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baabc41-821a-4519-85b2-dcbb00fd979c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:10.387071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fce4614-d2ea-4ced-9fe8-d4cdfafb168f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:10.387895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fce4614-d2ea-4ced-9fe8-d4cdfafb168f service placement] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:10.388335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fce4614-d2ea-4ced-9fe8-d4cdfafb168f service placement] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:10.388829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fce4614-d2ea-4ced-9fe8-d4cdfafb168f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:10.403491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1fce4614-d2ea-4ced-9fe8-d4cdfafb168f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:10.420735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fce4614-d2ea-4ced-9fe8-d4cdfafb168f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:10.422444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fce4614-d2ea-4ced-9fe8-d4cdfafb168f service placement] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:10.432849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1159/2266] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:14.998223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:15.008151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b893ccc-b80e-450a-9850-ca95ea2350c5 service swift] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:15.011743 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:15.012359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:15.012635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:15.012906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:15.014665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:15.078254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:15.079845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:15.090113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1108/2267] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:14 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:20.314478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43216649-ec5d-445c-aed2-847719e30c7f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:20.316240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43216649-ec5d-445c-aed2-847719e30c7f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:20.316772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43216649-ec5d-445c-aed2-847719e30c7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:20.347636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-43216649-ec5d-445c-aed2-847719e30c7f None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:20.350428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-43216649-ec5d-445c-aed2-847719e30c7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:20.364270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1160/2268] 10.222.0.27 () {58 vars in 967 bytes} [Tue Oct 4 08:55:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:20.379042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:20.404505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2bc01f-27c1-454c-b712-0998b60b90d0 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:20.406665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6187cd40-a288-49e1-acbb-f696d09cca42 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:20.407684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6187cd40-a288-49e1-acbb-f696d09cca42 service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:20.408094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6187cd40-a288-49e1-acbb-f696d09cca42 service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:20.409079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6187cd40-a288-49e1-acbb-f696d09cca42 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:20.419446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6187cd40-a288-49e1-acbb-f696d09cca42 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:20.421104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6187cd40-a288-49e1-acbb-f696d09cca42 service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:20.451552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1109/2269] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:55:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:55:30.338617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=35 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:30.437385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=36 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:33.227362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dbb6a4-b9b3-4945-8918-1ad9ef12c49e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.227872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dbb6a4-b9b3-4945-8918-1ad9ef12c49e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.227969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dbb6a4-b9b3-4945-8918-1ad9ef12c49e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.247814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d4dbb6a4-b9b3-4945-8918-1ad9ef12c49e None None] MFA Rules not processed for user `2677a4d76de24a729b87a211b49b965a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:33.249020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d4dbb6a4-b9b3-4945-8918-1ad9ef12c49e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:33.303631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1161/2270] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:55:33 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:43.297001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=25 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:35.067848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ba9c4-05b9-4ac2-a8d0-adc2dd645812 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:35.069038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ba9c4-05b9-4ac2-a8d0-adc2dd645812 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:35.069340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ba9c4-05b9-4ac2-a8d0-adc2dd645812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:35.091774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f32ba9c4-05b9-4ac2-a8d0-adc2dd645812 None None] MFA Rules not processed for user `14d56042415540a3b3ed06578b712ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:35.107158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f32ba9c4-05b9-4ac2-a8d0-adc2dd645812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:35.170810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1110/2271] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:56:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:36.601073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:36.601606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:36.601606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:36.620989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] MFA Rules not processed for user `835a297b6e75416ea9bd1a82b466b2a9`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:36.622207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:36.700099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1162/2272] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:56:36 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:45.163880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=25 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:46.693499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=25 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:47.096373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af14f1ac-218a-4083-ba81-e2ba2aadf6b3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:47.096891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af14f1ac-218a-4083-ba81-e2ba2aadf6b3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:47.096891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af14f1ac-218a-4083-ba81-e2ba2aadf6b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:47.117686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-af14f1ac-218a-4083-ba81-e2ba2aadf6b3 None None] MFA Rules not processed for user `185e359e571f44bda1585124529b36a8`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:47.118929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-af14f1ac-218a-4083-ba81-e2ba2aadf6b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:47.176738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1111/2273] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:56:47 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:52.442008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:52.444347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:52.554006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144aa33a-3696-4b7d-88cd-e9fed6efca01 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:52.558870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:52.559519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:52.559965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:52.560392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] PATH_INFO: `/v3/users/2677a4d76de24a729b87a211b49b965a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:52.561685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2677a4d76de24a729b87a211b49b965a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:52.564009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:52.790435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2677a4d76de24a729b87a211b49b965a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:52.813058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2677a4d76de24a729b87a211b49b965a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:52.814079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2677a4d76de24a729b87a211b49b965a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:52.815242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2677a4d76de24a729b87a211b49b965a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:52.905919 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2677a4d76de24a729b87a211b49b965a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:52.934012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1163/2274] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:56:52 2022] DELETE /identity/v3/users/2677a4d76de24a729b87a211b49b965a => generated 0 bytes in 492 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:53.311792 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af14f1ac-218a-4083-ba81-e2ba2aadf6b3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:53.329008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af14f1ac-218a-4083-ba81-e2ba2aadf6b3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:53.400600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:53.401472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:53.401753 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:53.402060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] PATH_INFO: `/v3/projects/b1f2fb36dae5483ebc61c88336461b18` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:53.403415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1f2fb36dae5483ebc61c88336461b18)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:53.405347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:53.523823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1f2fb36dae5483ebc61c88336461b18', 'request_id': 'req-1757445e-461f-4b9c-ae90-7c8de0ab0f45'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:53.624640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1f2fb36dae5483ebc61c88336461b18', 'request_id': 'req-1757445e-461f-4b9c-ae90-7c8de0ab0f45'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:53.632286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1f2fb36dae5483ebc61c88336461b18', 'request_id': 'req-1757445e-461f-4b9c-ae90-7c8de0ab0f45'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:53.643379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] The token cache is being invalidate because project b1f2fb36dae5483ebc61c88336461b18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:56:53.643892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:53.644552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:53.646195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1112/2275] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:56:53 2022] DELETE /identity/v3/projects/b1f2fb36dae5483ebc61c88336461b18 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:56.117233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:56.118673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:56.177203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138e3106-8818-4e49-bd20-f6c90b0d1ea2 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:56.209794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:56.226626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b9bc46-2763-4575-b54a-1a3e2b011cce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:56.227249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b9bc46-2763-4575-b54a-1a3e2b011cce service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:56.227564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b9bc46-2763-4575-b54a-1a3e2b011cce service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:56.227870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b9bc46-2763-4575-b54a-1a3e2b011cce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:56.229697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-64b9bc46-2763-4575-b54a-1a3e2b011cce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:56.301671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b9bc46-2763-4575-b54a-1a3e2b011cce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:56.304066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b9bc46-2763-4575-b54a-1a3e2b011cce service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:56.320140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1164/2276] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:56:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:56:56.491035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:56.492748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:56.508072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1757445e-461f-4b9c-ae90-7c8de0ab0f45 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:56.512205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:56.512822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:56.513099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:56.513367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:56.515128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:56.530337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:56.531804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:56.545477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1113/2277] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:56:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:06.425864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=38 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:06.541130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=82,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:28.671391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:28.671911 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:28.672009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:28.692072 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:28.710125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:28.747775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1165/2278] 10.222.0.27 () {58 vars in 967 bytes} [Tue Oct 4 08:57:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:28.938146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:28.939610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:29.011220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce4a27-d538-4ff4-b273-1253ad8c1bb0 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:29.016156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:29.016761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:29.017038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:29.017305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:29.026027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:29.027640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:29.035800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1114/2279] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:57:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:57:32.855226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:32.856808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:32.957148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ffcb39-6622-4d0e-af4e-0481cbdf72d3 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:32.961243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:32.961950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:32.963040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:32.963373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:32.965935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:33.057407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:33.059030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:33.073492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1166/2280] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:32 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:33.468267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:33.470355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:33.565809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34340b0-34f4-4b14-981a-f8455f2f8c41 service swift] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:33.565809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:33.565809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:33.565809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:33.566263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:33.566263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:33.573662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:33.575287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:33.585286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1115/2281] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:37.049881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:37.049881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:37.113419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e381747c-d1de-4aef-a7cf-0d56e03b1292 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:37.117897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:37.118661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:37.119080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:37.119346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] PATH_INFO: `/v3/users/21628ccf9dfb4ad1921aaf27712031c0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:37.120421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=21628ccf9dfb4ad1921aaf27712031c0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:37.122344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:37.123857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:37.302159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21628ccf9dfb4ad1921aaf27712031c0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.309942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21628ccf9dfb4ad1921aaf27712031c0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.310663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21628ccf9dfb4ad1921aaf27712031c0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.311271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21628ccf9dfb4ad1921aaf27712031c0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.396673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21628ccf9dfb4ad1921aaf27712031c0'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.414632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1167/2282] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:57:37 2022] DELETE /identity/v3/users/21628ccf9dfb4ad1921aaf27712031c0 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:37.633325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:37.647712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f40de7-0761-45ee-94c2-5e627db8da9a service placement] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:37.704791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:37.705420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:37.705720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:37.705987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] PATH_INFO: `/v3/projects/f6f4de2e638d4746a56a2d2023d2401b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:37.707117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6f4de2e638d4746a56a2d2023d2401b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:37.709091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:37.731091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:37.791143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6f4de2e638d4746a56a2d2023d2401b', 'request_id': 'req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.836420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6f4de2e638d4746a56a2d2023d2401b', 'request_id': 'req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.842935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6f4de2e638d4746a56a2d2023d2401b', 'request_id': 'req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.853662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] The token cache is being invalidate because project f6f4de2e638d4746a56a2d2023d2401b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:57:37.853950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.854619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:37.856130 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1116/2283] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:57:37 2022] DELETE /identity/v3/projects/f6f4de2e638d4746a56a2d2023d2401b => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:37.860092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:37.861513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:37.875118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c15504-eee8-4c1b-83c6-e1ed5d63f0ab admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:37.929622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:37.930246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:37.930566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:37.930843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] PATH_INFO: `/v3/users/e185df1688f245d098b15c9298e4b5a7` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:37.932605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e185df1688f245d098b15c9298e4b5a7)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:37.934553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:38.154622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e185df1688f245d098b15c9298e4b5a7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.162015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e185df1688f245d098b15c9298e4b5a7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.162752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e185df1688f245d098b15c9298e4b5a7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.163412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e185df1688f245d098b15c9298e4b5a7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.259842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbdd6cb9-7e56-496b-8597-78b250d413c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e185df1688f245d098b15c9298e4b5a7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.273521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:38.279423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1168/2284] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:57:37 2022] DELETE /identity/v3/users/e185df1688f245d098b15c9298e4b5a7 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:38.597340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:38.598931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:38.614270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30e1a4-f4a0-432c-8b4d-e47ba224cdbb admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:38.672368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:38.673095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:38.673276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:38.673552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] PATH_INFO: `/v3/projects/73f71a9ac36e40c38b0e1e7306da9e6d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:38.674703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] RBAC: Authorizing `identity:delete_project(project_id=73f71a9ac36e40c38b0e1e7306da9e6d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:38.676595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:38.785991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73f71a9ac36e40c38b0e1e7306da9e6d', 'request_id': 'req-4304d85e-c886-41d3-b236-1e8478d6245a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.859176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73f71a9ac36e40c38b0e1e7306da9e6d', 'request_id': 'req-4304d85e-c886-41d3-b236-1e8478d6245a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.866077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73f71a9ac36e40c38b0e1e7306da9e6d', 'request_id': 'req-4304d85e-c886-41d3-b236-1e8478d6245a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.878528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] The token cache is being invalidate because project 73f71a9ac36e40c38b0e1e7306da9e6d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:57:38.878839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.879462 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4304d85e-c886-41d3-b236-1e8478d6245a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:38.880977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1117/2285] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:57:38 2022] DELETE /identity/v3/projects/73f71a9ac36e40c38b0e1e7306da9e6d => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:38.887368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c9a73-0043-41de-bb02-005e264515bb None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:38.887669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c9a73-0043-41de-bb02-005e264515bb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:38.887942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06c9a73-0043-41de-bb02-005e264515bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:38.907787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f06c9a73-0043-41de-bb02-005e264515bb None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:38.908989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f06c9a73-0043-41de-bb02-005e264515bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:38.968806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1169/2286] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:57:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:39.038520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914beddc-79a5-4d79-a11a-8d0a9999da6e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.038866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914beddc-79a5-4d79-a11a-8d0a9999da6e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.039193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914beddc-79a5-4d79-a11a-8d0a9999da6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.059998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-914beddc-79a5-4d79-a11a-8d0a9999da6e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:39.061324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-914beddc-79a5-4d79-a11a-8d0a9999da6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:39.070812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1118/2287] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:57:39.080354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06c9a73-0043-41de-bb02-005e264515bb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:39.090107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06c9a73-0043-41de-bb02-005e264515bb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:39.094764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:39.095392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.095647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.095917 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.096568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:57:39.096868 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:39.098492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:39.110486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1170/2288] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:57:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:57:39.114744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914beddc-79a5-4d79-a11a-8d0a9999da6e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:39.124945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914beddc-79a5-4d79-a11a-8d0a9999da6e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:39.130209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:39.130873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.131146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.131413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.132108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:39.134682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:39.283348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1119/2289] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:57:39 2022] POST /identity/v3/projects => generated 373 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:57:39.287798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:39.298467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b0fbc1-cf1d-4b01-a5b2-9dc15627dd4f admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:39.355944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:39.356550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.356927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.357203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.357826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:39.359793 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:39.442486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1171/2290] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:57:39 2022] POST /identity/v3/users => generated 287 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:57:39.446576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:39.457107 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecec1405-932c-4562-8fb2-198a289cbf27 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:39.464174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:39.464785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.465095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.465363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.466019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:39.467640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:39.473145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:39.478326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1120/2291] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:57:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:57:39.482900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:39.493045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb71aec3-8e60-4908-a5d4-2ba4da0fdf7d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:39.498815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93bcb8-8377-4a36-8d15-ba3a4be8b7c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:39.499415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93bcb8-8377-4a36-8d15-ba3a4be8b7c0 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.499694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93bcb8-8377-4a36-8d15-ba3a4be8b7c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.499996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93bcb8-8377-4a36-8d15-ba3a4be8b7c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.500640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b93bcb8-8377-4a36-8d15-ba3a4be8b7c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:39.502231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b93bcb8-8377-4a36-8d15-ba3a4be8b7c0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:39.513072 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1172/2292] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:57:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:57:39.517060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:39.526959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e7be4-5e86-4d9d-a5af-6f2485ac515c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:39.531816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:39.532509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.533365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.533813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] PATH_INFO: `/v3/projects/655f83bbd3d847f3b060af2ff5250585/users/faa4a1f5f6954d92830b7df74952161e/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.550230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=655f83bbd3d847f3b060af2ff5250585, user_id=faa4a1f5f6954d92830b7df74952161e, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:39.552938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:39.631253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1121/2293] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:57:39 2022] PUT /identity/v3/projects/655f83bbd3d847f3b060af2ff5250585/users/faa4a1f5f6954d92830b7df74952161e/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:39.636957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.637242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.637511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.656912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:39.658155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:39.722995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1173/2294] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:39.733570 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:39.735375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:39.794079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d62c5e-1152-49b4-9593-7bcce0d14fd1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:39.797800 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:39.798428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:39.798717 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:39.798988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:39.808547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:39.809968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:39.819230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1122/2295] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:42.452773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:42.455556 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:42.469908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:42.600429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e3d766-bee4-4ecf-a135-ff01de1e4a88 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:42.616176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:42.616829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:42.617114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:42.617390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:42.619728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:42.732266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:42.744439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:42.758118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1174/2296] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:43.102219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:43.133384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ce1080-465a-4e1d-9149-9a1532654a80 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:43.154742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85a289-ab7e-400c-9088-b1944b9d4832 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:43.155395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85a289-ab7e-400c-9088-b1944b9d4832 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:43.155675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85a289-ab7e-400c-9088-b1944b9d4832 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:43.155983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85a289-ab7e-400c-9088-b1944b9d4832 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:43.161140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4d85a289-ab7e-400c-9088-b1944b9d4832 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:43.197097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85a289-ab7e-400c-9088-b1944b9d4832 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:43.199370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85a289-ab7e-400c-9088-b1944b9d4832 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:43.234313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1123/2297] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:43.300497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:43.308645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:43.334461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78574b-cedb-44a1-bc8f-5743ddcace1c service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:43.357186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:43.357820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:43.358096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:43.358361 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:43.363337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:43.502373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:43.504269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:43.519327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1175/2298] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:43 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:47.768217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4618037-6ae3-4528-b698-d99de25b9658 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:47.768711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4618037-6ae3-4528-b698-d99de25b9658 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:47.768949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4618037-6ae3-4528-b698-d99de25b9658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:47.805809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c4618037-6ae3-4528-b698-d99de25b9658 None None] MFA Rules not processed for user `faa4a1f5f6954d92830b7df74952161e`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:47.809060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c4618037-6ae3-4528-b698-d99de25b9658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:47.912532 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1124/2299] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:57:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:47.917159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:47.929119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c051585-5796-4203-806d-c783fe337b56 service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:47.934260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:47.935215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:47.935487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:47.935907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:47.936736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:47.938776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:48.061546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1176/2300] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:57:47 2022] POST /identity/v3/projects => generated 373 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:57:48.065670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4618037-6ae3-4528-b698-d99de25b9658 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:48.077507 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4618037-6ae3-4528-b698-d99de25b9658 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:48.156646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:48.164273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:48.164856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:48.165138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:48.165411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:48.166059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:48.169188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:48.289652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1125/2301] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:57:48 2022] POST /identity/v3/users => generated 293 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:57:48.295247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:48.310950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26a4ad8-ab1b-4321-bd32-d4236f441395 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:48.318349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:48.319038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:48.319741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:48.320012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:48.320698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:48.322792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:48.342813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1177/2302] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:57:48 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:57:48.346804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:48.359927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dadb25-4c21-434a-9e38-13d1ff6ab025 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:48.374500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:48.374500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:48.374500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:48.374500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] PATH_INFO: `/v3/projects/9241113fa00f46e5859c039629bc1f3d/users/b641fda917cf44949c6d93f7e1946cce/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:48.399152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=9241113fa00f46e5859c039629bc1f3d, user_id=b641fda917cf44949c6d93f7e1946cce, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:48.408314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:48.500388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1126/2303] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:57:48 2022] PUT /identity/v3/projects/9241113fa00f46e5859c039629bc1f3d/users/b641fda917cf44949c6d93f7e1946cce/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:48.504308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:48.515763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c27d5a-ce7c-4444-8344-f67dfd04ca92 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:48.530932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:48.602219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fdab9f-fdc7-46bb-a368-4c14e609bc91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:48.602843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdab9f-fdc7-46bb-a368-4c14e609bc91 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:48.605421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdab9f-fdc7-46bb-a368-4c14e609bc91 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:48.605421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdab9f-fdc7-46bb-a368-4c14e609bc91 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:48.605421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fdab9f-fdc7-46bb-a368-4c14e609bc91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:48.606016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fdab9f-fdc7-46bb-a368-4c14e609bc91 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:48.617627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1178/2304] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:57:48 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:57:48.622925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:48.635025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af25c9a-4a4a-4dcf-b974-c5108edc37bd admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:48.642817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:48.643500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:48.643780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:48.644052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] PATH_INFO: `/v3/projects/9241113fa00f46e5859c039629bc1f3d/users/b641fda917cf44949c6d93f7e1946cce/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:48.646262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9241113fa00f46e5859c039629bc1f3d, user_id=b641fda917cf44949c6d93f7e1946cce, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:48.648879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:48.661514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1127/2305] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:57:48 2022] PUT /identity/v3/projects/9241113fa00f46e5859c039629bc1f3d/users/b641fda917cf44949c6d93f7e1946cce/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:50.062175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:50.062790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:50.063007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:50.083492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] MFA Rules not processed for user `cb93504f2605489eaf50cf10930b3e46`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:50.084719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:50.157664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1179/2306] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:57:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:50.662785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:50.758573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7588e2-9a84-4eaa-a692-fb32e5c21d74 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:50.764313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94477429-7546-410d-8a2e-23caeb1e7d18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:50.765133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94477429-7546-410d-8a2e-23caeb1e7d18 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:50.765577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94477429-7546-410d-8a2e-23caeb1e7d18 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:50.766087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94477429-7546-410d-8a2e-23caeb1e7d18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:50.768513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-94477429-7546-410d-8a2e-23caeb1e7d18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:50.781047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94477429-7546-410d-8a2e-23caeb1e7d18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:50.783101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94477429-7546-410d-8a2e-23caeb1e7d18 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:50.798103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1128/2307] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:50 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:51.414893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:51.416520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:51.559766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825f4ae2-c1a2-4fdc-89d9-63b8324a57a1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:51.569742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:51.570422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:51.571012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:51.571132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:51.574481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:51.590024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:51.591747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:51.605843 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1180/2308] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:56.442885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:56.443448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:56.443448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:56.474424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] MFA Rules not processed for user `b641fda917cf44949c6d93f7e1946cce`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:56.475704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:56.554503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1129/2309] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:57:56 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:56.562694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:56.607902 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:56.637009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d85519-5b8c-48a6-ac71-66523759a42b service placement] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:56.642625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071605eb-05e8-471b-818a-be4af26d9ad0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:56.643237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071605eb-05e8-471b-818a-be4af26d9ad0 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:56.643518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071605eb-05e8-471b-818a-be4af26d9ad0 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:56.643788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071605eb-05e8-471b-818a-be4af26d9ad0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:56.700246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071605eb-05e8-471b-818a-be4af26d9ad0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:56.701757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071605eb-05e8-471b-818a-be4af26d9ad0 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:56.710727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1181/2310] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:56 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:57.887275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:57.888895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:57.902147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d22c71-84fc-4197-aa8e-f5fdd8355a18 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:57.906806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:57.907416 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:57.907840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:57.908151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:57.910632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:57.923005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:57.924445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:57.935480 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1130/2311] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:59.653597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e01683-759e-4905-b5d6-245c710ead0a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:59.654518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e01683-759e-4905-b5d6-245c710ead0a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:59.655067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e01683-759e-4905-b5d6-245c710ead0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:59.679709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a3e01683-759e-4905-b5d6-245c710ead0a None None] MFA Rules not processed for user `faa4a1f5f6954d92830b7df74952161e`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:59.681003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a3e01683-759e-4905-b5d6-245c710ead0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:59.689035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1182/2312] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:57:59 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:02.950594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:02.992584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa5f6af-0312-4d38-8802-01befdcddb9f service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:03.019610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f079179-db81-4337-bb10-bbbbc2ad247e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:03.020324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f079179-db81-4337-bb10-bbbbc2ad247e service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:03.020610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f079179-db81-4337-bb10-bbbbc2ad247e service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:03.020885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f079179-db81-4337-bb10-bbbbc2ad247e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:03.022963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5f079179-db81-4337-bb10-bbbbc2ad247e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:03.053153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f079179-db81-4337-bb10-bbbbc2ad247e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:03.059948 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f079179-db81-4337-bb10-bbbbc2ad247e service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:03.099903 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1131/2313] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:03.253664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e01683-759e-4905-b5d6-245c710ead0a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:03.272166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e01683-759e-4905-b5d6-245c710ead0a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:03.275949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:03.277164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:03.277442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:03.277748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:03.279698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:03.359374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:03.360390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:03.371383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1183/2314] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:03.645834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:03.646119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:03.646453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:03.670131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:03.671420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:03.676409 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:03.754608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1132/2315] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:58:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:03.787852 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:03.789382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:03.932460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:03.940021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a96a2-021a-4cf1-89c1-0834ef4757f8 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:03.946623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:03.949467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:03.949763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:03.950130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:03.959470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:03.966823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:03.971523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1184/2316] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:58:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:58:10.802094 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:10.806451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:10.834957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bee27bf-8c27-471c-9b64-8e0761c4a140 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:10.840072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:10.841411 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:10.841846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:10.842013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] PATH_INFO: `/v3/users/185e359e571f44bda1585124529b36a8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:10.844811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] RBAC: Authorizing `identity:delete_user(user_id=185e359e571f44bda1585124529b36a8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:10.852888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:11.107592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '185e359e571f44bda1585124529b36a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:11.117598 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '185e359e571f44bda1585124529b36a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:11.118340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '185e359e571f44bda1585124529b36a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:11.118975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '185e359e571f44bda1585124529b36a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:11.247562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1814fade-3d3f-48b3-a613-2d5a068b9fdc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '185e359e571f44bda1585124529b36a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:11.284119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1133/2317] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:58:10 2022] DELETE /identity/v3/users/185e359e571f44bda1585124529b36a8 => generated 0 bytes in 484 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:11.918269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:11.929506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fae858d-17cd-4cb2-ae9c-13b628e61afe service swift] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:12.013614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:12.014280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:12.014727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:12.014998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] PATH_INFO: `/v3/projects/24b4590edfe642bc8a8bd15a1b8ba464` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:12.016156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=24b4590edfe642bc8a8bd15a1b8ba464)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:12.018257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:12.181301 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24b4590edfe642bc8a8bd15a1b8ba464', 'request_id': 'req-b34fcb2d-9624-468d-9203-69a1eab888ba'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:12.194214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24b4590edfe642bc8a8bd15a1b8ba464', 'request_id': 'req-b34fcb2d-9624-468d-9203-69a1eab888ba'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:12.385057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24b4590edfe642bc8a8bd15a1b8ba464', 'request_id': 'req-b34fcb2d-9624-468d-9203-69a1eab888ba'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:12.404254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f5d5da-378d-47a6-8a02-7ac69200ab13 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:12.404783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f5d5da-378d-47a6-8a02-7ac69200ab13 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:12.405187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f5d5da-378d-47a6-8a02-7ac69200ab13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:12.423466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] The token cache is being invalidate because project 24b4590edfe642bc8a8bd15a1b8ba464 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:58:12.425665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:12.428193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b34fcb2d-9624-468d-9203-69a1eab888ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:12.436952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1185/2318] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:58:11 2022] DELETE /identity/v3/projects/24b4590edfe642bc8a8bd15a1b8ba464 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:12.442248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392e19b2-220f-47a7-abfa-b77fbe340276 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:12.442571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392e19b2-220f-47a7-abfa-b77fbe340276 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:12.442844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392e19b2-220f-47a7-abfa-b77fbe340276 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:12.464639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-94f5d5da-378d-47a6-8a02-7ac69200ab13 None None] MFA Rules not processed for user `14d56042415540a3b3ed06578b712ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:12.465888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-94f5d5da-378d-47a6-8a02-7ac69200ab13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:12.486967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-392e19b2-220f-47a7-abfa-b77fbe340276 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:12.488237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-392e19b2-220f-47a7-abfa-b77fbe340276 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:12.558431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1134/2319] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:58:12.578447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1186/2320] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:12.586915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e257f-3637-4409-aee9-11917c5d7336 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:12.587520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e257f-3637-4409-aee9-11917c5d7336 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:12.588576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e257f-3637-4409-aee9-11917c5d7336 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:12.646331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c13e257f-3637-4409-aee9-11917c5d7336 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:12.647865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c13e257f-3637-4409-aee9-11917c5d7336 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:12.659962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1135/2321] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:12.664418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392e19b2-220f-47a7-abfa-b77fbe340276 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:12.704915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392e19b2-220f-47a7-abfa-b77fbe340276 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:12.716152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:12.717002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:12.717346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:12.717681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:12.718456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:58:12.718826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:12.720641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:12.744419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1187/2322] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:58:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:58:12.744854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13e257f-3637-4409-aee9-11917c5d7336 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:12.763910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13e257f-3637-4409-aee9-11917c5d7336 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:12.779995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:12.782603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:12.782603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:12.782603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:12.784602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:12.790823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:12.946460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1136/2323] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:58:12 2022] POST /identity/v3/projects => generated 373 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:58:12.949865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:12.964515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2aab6-a949-44f8-bf14-305d7007435a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:12.986879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:13.105170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:13.106548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:13.106548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:13.106548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:13.107229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:13.111803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:13.140977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:13.143192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:13.150382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:13.210592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633580df-6186-407e-9352-0268cea27ef4 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:13.215882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:13.216980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:13.216980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:13.217225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:13.219273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:13.231770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:13.233236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:13.243591 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1137/2324] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:58:13.541273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1188/2325] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:58:12 2022] POST /identity/v3/users => generated 287 bytes in 594 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:58:13.545476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:13.560693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19d132-6ca3-4747-9f05-6c4423f376f6 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:13.565599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:13.578085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:13.578366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:13.578673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:13.579322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:13.581016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:13.607263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1138/2326] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:13 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:58:13.611440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:13.622670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa36f6-319d-4112-a882-f4c366bc6511 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:13.627046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca973e4-0d0c-4e64-8eb2-a2e89c636df0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:13.629225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca973e4-0d0c-4e64-8eb2-a2e89c636df0 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:13.629500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca973e4-0d0c-4e64-8eb2-a2e89c636df0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:13.629773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca973e4-0d0c-4e64-8eb2-a2e89c636df0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:13.630437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca973e4-0d0c-4e64-8eb2-a2e89c636df0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:13.632047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca973e4-0d0c-4e64-8eb2-a2e89c636df0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:13.644181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1189/2327] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:13 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:58:13.649561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:13.687994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3fb46-0abf-460f-870a-029514e4550b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:13.704333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:13.705244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:13.706165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:13.706631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] PATH_INFO: `/v3/projects/6d2655bf31d04940a66d895a03f1d853/users/b0abd1294d0d4309be4b1f9734f9ce20/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:13.728020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d2655bf31d04940a66d895a03f1d853, user_id=b0abd1294d0d4309be4b1f9734f9ce20, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:13.731967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:13.849009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1139/2328] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:58:13 2022] PUT /identity/v3/projects/6d2655bf31d04940a66d895a03f1d853/users/b0abd1294d0d4309be4b1f9734f9ce20/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:58:13.875690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3ec2e-458d-44c5-aee9-b72692201f6f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:13.876186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3ec2e-458d-44c5-aee9-b72692201f6f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:13.876492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3ec2e-458d-44c5-aee9-b72692201f6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:13.944459 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-57b3ec2e-458d-44c5-aee9-b72692201f6f None None] MFA Rules not processed for user `b0abd1294d0d4309be4b1f9734f9ce20`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:13.946351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-57b3ec2e-458d-44c5-aee9-b72692201f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:14.090024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1190/2329] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:14.099869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:14.105374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:14.205934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee40e0b-f0bb-42ce-b4dc-d29cabd957ae admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:14.211960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b95a16f-2790-4a08-83e5-e72de7e6c63c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:14.212777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b95a16f-2790-4a08-83e5-e72de7e6c63c service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:14.213059 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b95a16f-2790-4a08-83e5-e72de7e6c63c service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:14.213327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b95a16f-2790-4a08-83e5-e72de7e6c63c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:14.222654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b95a16f-2790-4a08-83e5-e72de7e6c63c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:14.224655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b95a16f-2790-4a08-83e5-e72de7e6c63c service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:14.235514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1140/2330] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:18.065648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b3ec2e-458d-44c5-aee9-b72692201f6f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:18.172562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b3ec2e-458d-44c5-aee9-b72692201f6f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:18.178620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:18.179355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:18.179748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:18.180088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] PATH_INFO: `/v3/users/b0abd1294d0d4309be4b1f9734f9ce20` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:18.181363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0abd1294d0d4309be4b1f9734f9ce20)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:18.184081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:18.215972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:18.477064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0abd1294d0d4309be4b1f9734f9ce20'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:18.488729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0abd1294d0d4309be4b1f9734f9ce20'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:18.489463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0abd1294d0d4309be4b1f9734f9ce20'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:18.490077 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0abd1294d0d4309be4b1f9734f9ce20'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:18.579409 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0abd1294d0d4309be4b1f9734f9ce20'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:18.614262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1191/2331] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:58:18 2022] DELETE /identity/v3/users/b0abd1294d0d4309be4b1f9734f9ce20 => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:18.618109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c148e386-f813-4b83-922f-97f97940b42b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:18.618686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c148e386-f813-4b83-922f-97f97940b42b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:18.618891 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c148e386-f813-4b83-922f-97f97940b42b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:18.644170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c148e386-f813-4b83-922f-97f97940b42b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:18.645762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c148e386-f813-4b83-922f-97f97940b42b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:18.716684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:18.733871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1141/2332] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:18.763620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:18.765279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:18.892492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2adda-73e7-44a2-b7b8-3bc521cbcf32 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:18.897069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1288dd13-5330-472d-8adc-79100832a119 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:18.897729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288dd13-5330-472d-8adc-79100832a119 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:18.898009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288dd13-5330-472d-8adc-79100832a119 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:18.898288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1288dd13-5330-472d-8adc-79100832a119 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:18.919740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1288dd13-5330-472d-8adc-79100832a119 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:18.926371 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1288dd13-5330-472d-8adc-79100832a119 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:18.935475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1192/2333] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:19.543644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c148e386-f813-4b83-922f-97f97940b42b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:19.545217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c148e386-f813-4b83-922f-97f97940b42b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:19.560219 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c148e386-f813-4b83-922f-97f97940b42b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:19.564971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:19.565702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:19.566062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:19.566425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] PATH_INFO: `/v3/projects/6d2655bf31d04940a66d895a03f1d853` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:19.567612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d2655bf31d04940a66d895a03f1d853)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:19.569658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:19.676066 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d2655bf31d04940a66d895a03f1d853', 'request_id': 'req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:19.738716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d2655bf31d04940a66d895a03f1d853', 'request_id': 'req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:19.746906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d2655bf31d04940a66d895a03f1d853', 'request_id': 'req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:19.759071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] The token cache is being invalidate because project 6d2655bf31d04940a66d895a03f1d853 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:58:19.759505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:19.760177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6347c98c-ea02-40c3-84d1-b3a8a2a595bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:19.762209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1142/2334] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:58:19 2022] DELETE /identity/v3/projects/6d2655bf31d04940a66d895a03f1d853 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:19.767542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393ac5b-6ad4-4640-baf4-c7761e2ab032 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:19.768054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393ac5b-6ad4-4640-baf4-c7761e2ab032 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:19.768054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393ac5b-6ad4-4640-baf4-c7761e2ab032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:19.789884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-5393ac5b-6ad4-4640-baf4-c7761e2ab032 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:19.789884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5393ac5b-6ad4-4640-baf4-c7761e2ab032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:19.851274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1193/2335] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:19.859398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c029c7cc-36c7-44c3-9f29-41e9bcd341f5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:19.859762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c029c7cc-36c7-44c3-9f29-41e9bcd341f5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:19.860157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c029c7cc-36c7-44c3-9f29-41e9bcd341f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:19.880776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c029c7cc-36c7-44c3-9f29-41e9bcd341f5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:19.882109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c029c7cc-36c7-44c3-9f29-41e9bcd341f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:19.890492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1143/2336] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:19.894995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5393ac5b-6ad4-4640-baf4-c7761e2ab032 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:19.905328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5393ac5b-6ad4-4640-baf4-c7761e2ab032 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:19.909945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:19.910669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:19.911026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:19.911344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:19.912153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:58:19.912521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:19.914165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:19.926495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1194/2337] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:58:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:58:19.930736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c029c7cc-36c7-44c3-9f29-41e9bcd341f5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:19.940386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c029c7cc-36c7-44c3-9f29-41e9bcd341f5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:19.944980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:19.945574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:19.946019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:19.946294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:19.946950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:19.949325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:20.033255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1144/2338] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:58:19 2022] POST /identity/v3/projects => generated 365 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:58:20.037212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:20.048352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227f6d11-489f-4606-9a82-770d57eb1504 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:20.075223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:20.114344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:20.115199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:20.115569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:20.115771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:20.116936 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:20.119247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:20.249668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1195/2339] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:58:20 2022] POST /identity/v3/users => generated 283 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:58:20.250848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:20.262796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b6187a-a8fb-4d47-a63b-3c4fca5c28b2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:20.267632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:20.268238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:20.268513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:20.268780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:20.269432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:20.271089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:20.281710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1145/2340] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:58:20.285621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:20.296424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1296a37-c4c0-4967-8d92-61582e544274 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:20.301378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff90a07-51e6-4895-a808-669ed36f0b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:20.301965 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff90a07-51e6-4895-a808-669ed36f0b65 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:20.302241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff90a07-51e6-4895-a808-669ed36f0b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:20.302548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff90a07-51e6-4895-a808-669ed36f0b65 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:20.303356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff90a07-51e6-4895-a808-669ed36f0b65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:20.304821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff90a07-51e6-4895-a808-669ed36f0b65 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:20.316251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1196/2341] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:58:20.320617 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:20.332131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a021026d-9faf-41a9-b9f3-cd7d8d803e15 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:20.338009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:20.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:20.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:20.341352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] PATH_INFO: `/v3/projects/949c5e2936764ebeb4847d5ad6fa6af1/users/f5da1c9c527a461fae92ecbb0757b04e/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:20.356913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=949c5e2936764ebeb4847d5ad6fa6af1, user_id=f5da1c9c527a461fae92ecbb0757b04e, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:20.359645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:20.420137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1146/2342] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:58:20 2022] PUT /identity/v3/projects/949c5e2936764ebeb4847d5ad6fa6af1/users/f5da1c9c527a461fae92ecbb0757b04e/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:58:20.430663 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a721fafe-a6ae-4112-a51b-1042f87f4f0c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:20.430949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a721fafe-a6ae-4112-a51b-1042f87f4f0c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:20.431217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a721fafe-a6ae-4112-a51b-1042f87f4f0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:20.462061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a721fafe-a6ae-4112-a51b-1042f87f4f0c None None] MFA Rules not processed for user `f5da1c9c527a461fae92ecbb0757b04e`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:20.463426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a721fafe-a6ae-4112-a51b-1042f87f4f0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:20.538258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1197/2343] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:20 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:20.549715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:20.551615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:20.615806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd5be51-b547-49d8-950b-bf566e576e65 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:20.620710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70ef1e4-f026-4c9a-8167-679da4f11ac4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:20.621871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ef1e4-f026-4c9a-8167-679da4f11ac4 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:20.622257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ef1e4-f026-4c9a-8167-679da4f11ac4 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:20.622571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ef1e4-f026-4c9a-8167-679da4f11ac4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:20.631635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70ef1e4-f026-4c9a-8167-679da4f11ac4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:20.633120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70ef1e4-f026-4c9a-8167-679da4f11ac4 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:20.644023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1147/2344] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:20 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:58:26.925783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db51a80-6c2e-4536-afd4-86d21ab67d61 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:26.926363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db51a80-6c2e-4536-afd4-86d21ab67d61 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:26.926497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db51a80-6c2e-4536-afd4-86d21ab67d61 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:26.927999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1198/2345] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:58:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:58:26.934815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d389bab9-e3a8-4f4f-9007-0dc2b1e56cae None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:26.939011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d389bab9-e3a8-4f4f-9007-0dc2b1e56cae None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:26.939011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d389bab9-e3a8-4f4f-9007-0dc2b1e56cae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:26.967733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d389bab9-e3a8-4f4f-9007-0dc2b1e56cae None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:26.969221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d389bab9-e3a8-4f4f-9007-0dc2b1e56cae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:26.990522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:27.047877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1148/2346] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:58:26 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:58:27.060683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db51a80-6c2e-4536-afd4-86d21ab67d61 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:27.063754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6db51a80-6c2e-4536-afd4-86d21ab67d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:27.165928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db51a80-6c2e-4536-afd4-86d21ab67d61 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:27.170701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5029e1-9e70-43df-943c-42695321b435 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:27.171333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5029e1-9e70-43df-943c-42695321b435 service glance] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:27.171610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5029e1-9e70-43df-943c-42695321b435 service glance] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:27.171883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5029e1-9e70-43df-943c-42695321b435 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:27.183081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5029e1-9e70-43df-943c-42695321b435 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:27.184478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5029e1-9e70-43df-943c-42695321b435 service glance] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:27.196393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1199/2347] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:27.392514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba015670-1b72-438d-a615-815f20a7d599 None None] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:27.392900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba015670-1b72-438d-a615-815f20a7d599 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:27.393297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba015670-1b72-438d-a615-815f20a7d599 None None] PATH_INFO: `/` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:27.395557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1149/2348] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:58:27.403972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3807aa09-00eb-481d-b10b-b4927f3ad28f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:27.404333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3807aa09-00eb-481d-b10b-b4927f3ad28f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:27.404682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3807aa09-00eb-481d-b10b-b4927f3ad28f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:27.424926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-3807aa09-00eb-481d-b10b-b4927f3ad28f None None] MFA Rules not processed for user `df391d2bf7884538823f04db5564d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:27.426133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3807aa09-00eb-481d-b10b-b4927f3ad28f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:27.460366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:27.479685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1200/2349] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:58:27 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:27.491748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba015670-1b72-438d-a615-815f20a7d599 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:27.493664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ba015670-1b72-438d-a615-815f20a7d599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:27.507001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba015670-1b72-438d-a615-815f20a7d599 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:27.511072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bb5081-69ba-4aef-9d83-380ac031d60b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:27.511933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bb5081-69ba-4aef-9d83-380ac031d60b service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:27.512208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bb5081-69ba-4aef-9d83-380ac031d60b service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:27.512467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bb5081-69ba-4aef-9d83-380ac031d60b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:27.519760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bb5081-69ba-4aef-9d83-380ac031d60b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:27.521171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bb5081-69ba-4aef-9d83-380ac031d60b service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:27.529837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1150/2350] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:27.654136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78129cb-4044-4117-a132-b688ac222d65 None None] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:27.654561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78129cb-4044-4117-a132-b688ac222d65 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:27.654906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78129cb-4044-4117-a132-b688ac222d65 None None] PATH_INFO: `/` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:27.656355 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1201/2351] 10.222.0.27 () {54 vars in 921 bytes} [Tue Oct 4 08:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:58:27.661794 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9105216-2d2c-45c5-b162-b72a873f5173 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:27.662076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9105216-2d2c-45c5-b162-b72a873f5173 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:27.662349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9105216-2d2c-45c5-b162-b72a873f5173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:27.682934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c9105216-2d2c-45c5-b162-b72a873f5173 None None] MFA Rules not processed for user `40e12aa547b84ac38fd222a9f5f62ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:27.684120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c9105216-2d2c-45c5-b162-b72a873f5173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:27.693458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1151/2352] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Oct 4 08:58:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:27.704639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78129cb-4044-4117-a132-b688ac222d65 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:27.715432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78129cb-4044-4117-a132-b688ac222d65 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:27.718922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:27.719522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:27.719798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:27.720069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:27.730703 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:27.732122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:27.743074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1202/2353] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:30.348831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c42aff-91e9-49f5-afa9-9996555eb30f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:30.349306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c42aff-91e9-49f5-afa9-9996555eb30f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:30.349402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c42aff-91e9-49f5-afa9-9996555eb30f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:30.369712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-01c42aff-91e9-49f5-afa9-9996555eb30f None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:30.371491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-01c42aff-91e9-49f5-afa9-9996555eb30f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:30.426370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1152/2354] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 08:58:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:30.448621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:30.450055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:30.521088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5d5684-16d6-494f-9ada-17847c08d529 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:30.526038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97932314-c62c-40df-be58-b7a838060283 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:30.526682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97932314-c62c-40df-be58-b7a838060283 service swift] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:30.526962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97932314-c62c-40df-be58-b7a838060283 service swift] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:30.527236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97932314-c62c-40df-be58-b7a838060283 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:30.535669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97932314-c62c-40df-be58-b7a838060283 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:30.537271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97932314-c62c-40df-be58-b7a838060283 service swift] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:30.546940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1203/2355] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:58:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 08:58:30.590670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c42aff-91e9-49f5-afa9-9996555eb30f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:30.655272 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c42aff-91e9-49f5-afa9-9996555eb30f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:30.660924 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:30.661627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:30.662321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:30.662321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] PATH_INFO: `/v3/users/f5da1c9c527a461fae92ecbb0757b04e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:30.663476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5da1c9c527a461fae92ecbb0757b04e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:30.665517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:30.874264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:30.885650 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5da1c9c527a461fae92ecbb0757b04e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:30.896654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5da1c9c527a461fae92ecbb0757b04e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:30.898521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5da1c9c527a461fae92ecbb0757b04e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:30.899456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5da1c9c527a461fae92ecbb0757b04e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:30.959090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5da1c9c527a461fae92ecbb0757b04e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:31.008419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1153/2356] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:58:30 2022] DELETE /identity/v3/users/f5da1c9c527a461fae92ecbb0757b04e => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:31.015155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:31.015712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:31.019665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:31.040956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:31.042620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:31.172024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1204/2357] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:31.184782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:31.187445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:31.259428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c625fc-f03d-4fcb-bfdd-0b342eb3e3b0 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:31.264996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27b7391-e15f-4fae-adce-a5806bbbfafa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:31.265881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27b7391-e15f-4fae-adce-a5806bbbfafa service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:31.266172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27b7391-e15f-4fae-adce-a5806bbbfafa service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:31.266629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27b7391-e15f-4fae-adce-a5806bbbfafa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:31.281163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27b7391-e15f-4fae-adce-a5806bbbfafa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:31.283270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27b7391-e15f-4fae-adce-a5806bbbfafa service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:31.298351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1154/2358] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:31.763609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:31.765959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:31.780691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ed38f1-e79b-4848-95e2-2accd9020f56 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:31.786369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:31.793436 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:31.798976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:31.805032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] PATH_INFO: `/v3/projects/949c5e2936764ebeb4847d5ad6fa6af1` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:31.807153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=949c5e2936764ebeb4847d5ad6fa6af1)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:31.810385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:31.923665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '949c5e2936764ebeb4847d5ad6fa6af1', 'request_id': 'req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:31.930280 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '949c5e2936764ebeb4847d5ad6fa6af1', 'request_id': 'req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:31.936111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:31.990811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '949c5e2936764ebeb4847d5ad6fa6af1', 'request_id': 'req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:32.008058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] The token cache is being invalidate because project 949c5e2936764ebeb4847d5ad6fa6af1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:58:32.008390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:32.009105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70f1cb8f-5e73-4b98-be1c-4ea5fda08c7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:58:32.010694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1205/2359] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:58:31 2022] DELETE /identity/v3/projects/949c5e2936764ebeb4847d5ad6fa6af1 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:32.015904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a0c19-2a11-4c52-b249-3edda0f9d1db None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.016400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a0c19-2a11-4c52-b249-3edda0f9d1db None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.016481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a0c19-2a11-4c52-b249-3edda0f9d1db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.045807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-997a0c19-2a11-4c52-b249-3edda0f9d1db None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:32.047099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-997a0c19-2a11-4c52-b249-3edda0f9d1db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:32.116366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1155/2360] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:58:32.124650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4977a10a-c390-41ca-a33a-0784016fe2a6 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.124931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4977a10a-c390-41ca-a33a-0784016fe2a6 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.125203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4977a10a-c390-41ca-a33a-0784016fe2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.146681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-4977a10a-c390-41ca-a33a-0784016fe2a6 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:32.147969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4977a10a-c390-41ca-a33a-0784016fe2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:32.163831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1206/2361] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:32.168799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997a0c19-2a11-4c52-b249-3edda0f9d1db None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:32.185281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997a0c19-2a11-4c52-b249-3edda0f9d1db None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:32.190293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:32.190967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.191247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.191519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.192172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:58:32.192470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:32.194078 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:32.217611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1156/2362] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:58:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:58:32.217724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4977a10a-c390-41ca-a33a-0784016fe2a6 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:32.225689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4977a10a-c390-41ca-a33a-0784016fe2a6 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:32.250664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:32.251302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.251592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.251930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.252637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:32.254595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:32.376544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1207/2363] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:58:32 2022] POST /identity/v3/projects => generated 377 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:58:32.380786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:32.395671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fce3e6f-1512-460a-8121-765ccf659a08 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:32.442675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:32.458669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:32.461042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.461042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.461042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.461042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:32.462345 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:32.536179 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1157/2364] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:58:32 2022] POST /identity/v3/users => generated 289 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:58:32.540257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:32.552683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050846b1-81be-4774-b6d0-3b3dd4642a95 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:32.558015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:32.558816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.559194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.559621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.560376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:32.562101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:32.574802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1208/2365] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:58:32.578678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:32.589990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab09c0f-4ade-4395-b11f-9f6ce109e2fe admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:32.595562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7309355-6555-4612-ae66-80436dbdf5bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:32.596258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7309355-6555-4612-ae66-80436dbdf5bc admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.596668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7309355-6555-4612-ae66-80436dbdf5bc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.596938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7309355-6555-4612-ae66-80436dbdf5bc admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.597597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7309355-6555-4612-ae66-80436dbdf5bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:32.599237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7309355-6555-4612-ae66-80436dbdf5bc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:32.611951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1158/2366] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:58:32.616420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:32.628103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ee596-d4ec-4a92-bf02-281ffbc6addd admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:32.633419 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:32.634096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.634430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.634659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] PATH_INFO: `/v3/projects/2fddbc00ed57422fb6670a7e50be204d/users/e9aae08853e641bb88dd52ea1102445d/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.652130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fddbc00ed57422fb6670a7e50be204d, user_id=e9aae08853e641bb88dd52ea1102445d, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:32.654853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:32.732443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1209/2367] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:58:32 2022] PUT /identity/v3/projects/2fddbc00ed57422fb6670a7e50be204d/users/e9aae08853e641bb88dd52ea1102445d/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:32.743067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143201db-7449-49e1-be7b-df11d0ed9b97 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.743669 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143201db-7449-49e1-be7b-df11d0ed9b97 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.744127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143201db-7449-49e1-be7b-df11d0ed9b97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.775865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-143201db-7449-49e1-be7b-df11d0ed9b97 None None] MFA Rules not processed for user `e9aae08853e641bb88dd52ea1102445d`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:32.778160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-143201db-7449-49e1-be7b-df11d0ed9b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:32.857520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1159/2368] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:32.861953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:32.924711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5314b2b-2be9-4ae5-9235-2d9f7b979f7d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:32.929400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:32.930004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:32.930286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:32.930604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:32.931230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:32.934289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:33.060520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1210/2369] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:58:32 2022] POST /identity/v3/projects => generated 377 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:58:33.064569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143201db-7449-49e1-be7b-df11d0ed9b97 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:33.075798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143201db-7449-49e1-be7b-df11d0ed9b97 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:33.138890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:33.139427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:33.139705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:33.139967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:33.140736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:33.142692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:33.254119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1160/2370] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:58:33 2022] POST /identity/v3/users => generated 295 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:58:33.258615 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:33.269479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a0a972-18dc-4afc-8b1e-a9a67ab632e2 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:33.274060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:33.274719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:33.274996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:33.275266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:33.280453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:33.280453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:33.288080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1211/2371] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:58:33.292950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:33.303187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de8ae1-973f-4ac5-84db-655d4c8df9d2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:33.308361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:33.309144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:33.309426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:33.309703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] PATH_INFO: `/v3/projects/246b326c1ff0487c87e0d292d17c29aa/users/6082abdadef04826a385f52eaa1c2956/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:33.324928 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] RBAC: Authorizing `identity:create_grant(project_id=246b326c1ff0487c87e0d292d17c29aa, user_id=6082abdadef04826a385f52eaa1c2956, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:33.327642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:33.407699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1161/2372] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:58:33 2022] PUT /identity/v3/projects/246b326c1ff0487c87e0d292d17c29aa/users/6082abdadef04826a385f52eaa1c2956/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:33.413054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:33.423946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68a8e9b-cb7b-44ec-8e3e-5dcd4cf4da40 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:33.519962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:33.526506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088abd86-a7a1-485e-880f-4afb6b5ca4a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:33.527820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088abd86-a7a1-485e-880f-4afb6b5ca4a9 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:33.532078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088abd86-a7a1-485e-880f-4afb6b5ca4a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:33.532078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088abd86-a7a1-485e-880f-4afb6b5ca4a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:33.532078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088abd86-a7a1-485e-880f-4afb6b5ca4a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:33.532078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088abd86-a7a1-485e-880f-4afb6b5ca4a9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:33.550517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1212/2373] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:58:33.555333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:33.572504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5743bb1-255c-40fb-b69b-37bda1272713 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:33.581981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:33.583365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:33.583804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:33.584178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] PATH_INFO: `/v3/projects/246b326c1ff0487c87e0d292d17c29aa/users/6082abdadef04826a385f52eaa1c2956/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:33.587770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=246b326c1ff0487c87e0d292d17c29aa, user_id=6082abdadef04826a385f52eaa1c2956, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:33.592132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:33.614937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1162/2374] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:58:33 2022] PUT /identity/v3/projects/246b326c1ff0487c87e0d292d17c29aa/users/6082abdadef04826a385f52eaa1c2956/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:33.626736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:33.627491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:33.627930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:33.698727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] MFA Rules not processed for user `6082abdadef04826a385f52eaa1c2956`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:33.700080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:33.783518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1213/2375] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:58:33 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:33.795477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:33.797159 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:33.803579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:33.855944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b74a143-8fee-4c84-bca4-efb52c583e87 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:33.859757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:33.860341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:33.860653 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:33.860984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:33.933514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:33.935138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:33.958567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1163/2376] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:33 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:40.417563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:40.419206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:40.432013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae625b58-dbe8-4795-b2e3-b75adde331d7 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:40.436702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032bbe1d-3973-4c00-b212-cbdb6c38812e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:40.437386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032bbe1d-3973-4c00-b212-cbdb6c38812e service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:40.437670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032bbe1d-3973-4c00-b212-cbdb6c38812e service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:40.438121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032bbe1d-3973-4c00-b212-cbdb6c38812e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:40.450240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032bbe1d-3973-4c00-b212-cbdb6c38812e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:40.451730 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032bbe1d-3973-4c00-b212-cbdb6c38812e service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:40.463415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1214/2377] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:40 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:43.942176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=46 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:47.683317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:47.684761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:47.743248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c80a437-6a6a-466a-839c-30db4d20b237 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:47.746836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:47.747434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:47.747712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:47.748559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:47.750533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:47.812626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:47.813962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:47.823263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1164/2378] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:47 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:50.458370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=43 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:52.216216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:52.217891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:52.217891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:52.239518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:58:52.240733 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:52.301316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1215/2379] 10.222.0.27 () {58 vars in 967 bytes} [Tue Oct 4 08:58:52 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:58:52.314766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:52.319419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:52.384675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2408a84-b36f-4907-b4c4-5f4d960c22d5 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:52.389859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:52.390582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:52.390764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:52.391037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:52.399163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:52.400789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:52.409396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1165/2380] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 08:58:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:58:56.955799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:56.957187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:57.027221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9c91e0-5975-40df-8416-c9a3514eda74 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:57.032613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d55c3c9-d0c3-4339-a8c9-68af7e5cd648 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:57.033215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d55c3c9-d0c3-4339-a8c9-68af7e5cd648 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:57.033487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d55c3c9-d0c3-4339-a8c9-68af7e5cd648 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:57.033755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d55c3c9-d0c3-4339-a8c9-68af7e5cd648 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:57.035438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2d55c3c9-d0c3-4339-a8c9-68af7e5cd648 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:57.092716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d55c3c9-d0c3-4339-a8c9-68af7e5cd648 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:57.094322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d55c3c9-d0c3-4339-a8c9-68af7e5cd648 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:57.103405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1216/2381] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:02.406054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=82 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:07.099308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=81 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:07.520612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:07.582998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7346a-c453-4a62-a644-7913f792dbec service swift] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:07.587548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:07.588170 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:07.588446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:07.588716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] PATH_INFO: `/v3/users/e9aae08853e641bb88dd52ea1102445d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:07.589828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9aae08853e641bb88dd52ea1102445d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:07.591755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:07.741051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9aae08853e641bb88dd52ea1102445d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:07.748005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9aae08853e641bb88dd52ea1102445d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:07.748673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9aae08853e641bb88dd52ea1102445d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:07.749270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9aae08853e641bb88dd52ea1102445d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:07.812809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9aae08853e641bb88dd52ea1102445d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:07.831298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1166/2382] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:07 2022] DELETE /identity/v3/users/e9aae08853e641bb88dd52ea1102445d => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:07.836761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:07.837158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:07.837432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:07.857999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:07.859225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:07.919031 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1217/2383] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:07.928768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:07.930200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:07.986381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162330f3-dad3-4646-816c-c5adf8d97ca1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:07.989940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:07.990566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:07.990842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:07.991162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:07.999922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:08.001836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:08.011339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1167/2384] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:08.603670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:08.604986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:08.620792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cd3c35-dc91-4125-bd18-c4e21e033e22 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:08.627238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:08.628010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:08.628179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:08.628430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] PATH_INFO: `/v3/projects/2fddbc00ed57422fb6670a7e50be204d` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:08.629598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2fddbc00ed57422fb6670a7e50be204d)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:08.631559 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:08.717111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2fddbc00ed57422fb6670a7e50be204d', 'request_id': 'req-5abedb9d-3e86-45a6-8087-3958ebb2fc37'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:08.724190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2fddbc00ed57422fb6670a7e50be204d', 'request_id': 'req-5abedb9d-3e86-45a6-8087-3958ebb2fc37'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:08.767140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2fddbc00ed57422fb6670a7e50be204d', 'request_id': 'req-5abedb9d-3e86-45a6-8087-3958ebb2fc37'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:08.777493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] The token cache is being invalidate because project 2fddbc00ed57422fb6670a7e50be204d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:08.777778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:08.779799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:08.779947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1218/2385] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:08 2022] DELETE /identity/v3/projects/2fddbc00ed57422fb6670a7e50be204d => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:08.785426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:08.800529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696d3350-9c42-4f6c-b02f-0e16974acae6 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:08.824750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:08.863879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:08.864472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:08.864750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:08.865022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] PATH_INFO: `/v3/users/6082abdadef04826a385f52eaa1c2956` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:08.866157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6082abdadef04826a385f52eaa1c2956)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:08.868127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:09.010042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6082abdadef04826a385f52eaa1c2956'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.017249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6082abdadef04826a385f52eaa1c2956'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.017943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6082abdadef04826a385f52eaa1c2956'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.018571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6082abdadef04826a385f52eaa1c2956'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.058706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b8ba27fe-0ace-46ce-b24c-9add482c1768 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6082abdadef04826a385f52eaa1c2956'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.076217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1168/2386] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:08 2022] DELETE /identity/v3/users/6082abdadef04826a385f52eaa1c2956 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:09.529172 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:09.530906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:09.546005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abedb9d-3e86-45a6-8087-3958ebb2fc37 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:09.611154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:09.611879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:09.612138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:09.612358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] PATH_INFO: `/v3/projects/246b326c1ff0487c87e0d292d17c29aa` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:09.613493 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=246b326c1ff0487c87e0d292d17c29aa)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:09.615453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:09.740677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:09.757829 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '246b326c1ff0487c87e0d292d17c29aa', 'request_id': 'req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.764805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '246b326c1ff0487c87e0d292d17c29aa', 'request_id': 'req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.835228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '246b326c1ff0487c87e0d292d17c29aa', 'request_id': 'req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.846257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] The token cache is being invalidate because project 246b326c1ff0487c87e0d292d17c29aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:09.846749 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.847455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-70c72547-0a1a-448f-9e8e-c1e9ec6243eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:09.849299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1219/2387] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:09 2022] DELETE /identity/v3/projects/246b326c1ff0487c87e0d292d17c29aa => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:09.854499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd2b41-588a-47e8-9712-1642b30919ac None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:09.854997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd2b41-588a-47e8-9712-1642b30919ac None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:09.855149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd2b41-588a-47e8-9712-1642b30919ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:09.880481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-addd2b41-588a-47e8-9712-1642b30919ac None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:09.881703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-addd2b41-588a-47e8-9712-1642b30919ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:09.943605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1169/2388] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:09.951788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba322f9-e0bf-4aa0-9720-93a19d7b55eb None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:09.952073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba322f9-e0bf-4aa0-9720-93a19d7b55eb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:09.952345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba322f9-e0bf-4aa0-9720-93a19d7b55eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:09.973393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-aba322f9-e0bf-4aa0-9720-93a19d7b55eb None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:09.974856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-aba322f9-e0bf-4aa0-9720-93a19d7b55eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:09.982942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1220/2389] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:09.987688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd2b41-588a-47e8-9712-1642b30919ac None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:09.997258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd2b41-588a-47e8-9712-1642b30919ac None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.002875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.003490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.003770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.004172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.004893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:59:10.005119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.007236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.020659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1170/2390] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:59:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:59:10.024440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba322f9-e0bf-4aa0-9720-93a19d7b55eb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.035207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba322f9-e0bf-4aa0-9720-93a19d7b55eb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.039971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.040566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.040841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.043248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.043248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.043765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.127246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1221/2391] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:10 2022] POST /identity/v3/projects => generated 367 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:10.128897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.139389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0a0133-0090-471d-b455-52d588dbead7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.168337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:10.201432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.202044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.202323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.202631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.203267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.205228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.304722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1171/2392] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:10 2022] POST /identity/v3/users => generated 284 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:10.308951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.320509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1f86dd-cc23-4f17-8904-1baad9ee0e17 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.325227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.325824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.326102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.329716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.329716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.329716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.339182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1222/2393] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:10.343236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.353374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f0acb-e4d8-405b-9b9b-d2b97b7b311a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.358500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a78f16-4ac2-473c-8e4a-33516199eb1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.358728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a78f16-4ac2-473c-8e4a-33516199eb1f admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.358862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a78f16-4ac2-473c-8e4a-33516199eb1f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.359133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a78f16-4ac2-473c-8e4a-33516199eb1f admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.359802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a78f16-4ac2-473c-8e4a-33516199eb1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.361412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a78f16-4ac2-473c-8e4a-33516199eb1f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.371206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1172/2394] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:10 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:10.376271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.386975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b041ec9-ae8f-4f0e-af73-fd1549dd3d69 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.393006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.394005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.394284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.394596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] PATH_INFO: `/v3/projects/c91b5cfec8a943c19bfb5968dea35bea/users/e7ca1b5ec9e3475ab9f8c155ee1588bf/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.409593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c91b5cfec8a943c19bfb5968dea35bea, user_id=e7ca1b5ec9e3475ab9f8c155ee1588bf, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.412297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.493442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1223/2395] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:10 2022] PUT /identity/v3/projects/c91b5cfec8a943c19bfb5968dea35bea/users/e7ca1b5ec9e3475ab9f8c155ee1588bf/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:10.505729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f627b-ef3e-4ae0-b49f-73c6d918223b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.506510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f627b-ef3e-4ae0-b49f-73c6d918223b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.506510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f627b-ef3e-4ae0-b49f-73c6d918223b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.535038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7e5f627b-ef3e-4ae0-b49f-73c6d918223b None None] MFA Rules not processed for user `e7ca1b5ec9e3475ab9f8c155ee1588bf`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:10.536276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7e5f627b-ef3e-4ae0-b49f-73c6d918223b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:10.600415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1173/2396] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:10 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:10.604885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.666550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de2ee60-d7e2-4e0e-b6ad-a26f70dffd92 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.671516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.672688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.674711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.674711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.674711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.675802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.744639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1224/2397] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:10 2022] POST /identity/v3/projects => generated 369 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:10.746284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f627b-ef3e-4ae0-b49f-73c6d918223b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.765384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f627b-ef3e-4ae0-b49f-73c6d918223b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.832587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.833191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.836042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.836042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.836042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.836328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.933312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1174/2398] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:10 2022] POST /identity/v3/users => generated 291 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:10.939452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.948439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47949860-918a-4de5-a9d3-d12bb1eb9950 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.953018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.953680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.953996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.954233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:10.954980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:10.956579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:10.966841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1225/2399] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:10.971455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:10.981338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac38afe-1396-4326-bfac-8fd41f50bce2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:10.986853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:10.987497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:10.987769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:10.988037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] PATH_INFO: `/v3/projects/895f1b36689342429cc4f60730cd121e/users/0659b0d0222b4f7d8578f6c7665d2d0a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:11.003611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] RBAC: Authorizing `identity:create_grant(project_id=895f1b36689342429cc4f60730cd121e, user_id=0659b0d0222b4f7d8578f6c7665d2d0a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:11.006160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:11.086757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1175/2400] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:10 2022] PUT /identity/v3/projects/895f1b36689342429cc4f60730cd121e/users/0659b0d0222b4f7d8578f6c7665d2d0a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:11.090689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:11.101738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb59f1-7c2d-4498-b7f5-b19d9c17afa3 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:11.125210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:11.162536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da9422f-5393-489f-a1a6-1210e885d068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:11.163132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9422f-5393-489f-a1a6-1210e885d068 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:11.163407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9422f-5393-489f-a1a6-1210e885d068 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:11.163676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da9422f-5393-489f-a1a6-1210e885d068 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:11.164307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da9422f-5393-489f-a1a6-1210e885d068 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:11.165891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da9422f-5393-489f-a1a6-1210e885d068 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:11.176941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1226/2401] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:11 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:11.181117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:11.186762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:11.192461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6736d48b-af81-4a07-8572-4daa60bdc72d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:11.197217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:11.197853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:11.198125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:11.198449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] PATH_INFO: `/v3/projects/895f1b36689342429cc4f60730cd121e/users/0659b0d0222b4f7d8578f6c7665d2d0a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:11.200361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] RBAC: Authorizing `identity:create_grant(project_id=895f1b36689342429cc4f60730cd121e, user_id=0659b0d0222b4f7d8578f6c7665d2d0a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:11.203109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:11.214221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1176/2402] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:11 2022] PUT /identity/v3/projects/895f1b36689342429cc4f60730cd121e/users/0659b0d0222b4f7d8578f6c7665d2d0a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:11.225117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:11.225400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:11.225674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:11.255318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] MFA Rules not processed for user `0659b0d0222b4f7d8578f6c7665d2d0a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:11.256438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:11.323560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1227/2403] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:11 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:11.335907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:11.337609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:11.395761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b42d17-b414-42e7-ac3d-4801e890c17a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:11.400502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:11.401180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:11.401455 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:11.401721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:11.413862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:11.415436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:11.427930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1177/2404] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:11 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:11.513293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:11.514702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:11.526788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c8026c-233d-42c2-8556-f43f22f0b48e None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:11.534588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb4dc64-fe21-4379-af05-34979438f7c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:11.535189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb4dc64-fe21-4379-af05-34979438f7c5 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:11.535468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb4dc64-fe21-4379-af05-34979438f7c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:11.535739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb4dc64-fe21-4379-af05-34979438f7c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:11.590605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb4dc64-fe21-4379-af05-34979438f7c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:11.592057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb4dc64-fe21-4379-af05-34979438f7c5 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:11.601584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1228/2405] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:11 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:16.304981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:16.368604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ed8ab7-5387-4da7-a90a-077cfc10ba78 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:16.373537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:16.374313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:16.374612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:16.374988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] PATH_INFO: `/v3/users/e7ca1b5ec9e3475ab9f8c155ee1588bf` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:16.376766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7ca1b5ec9e3475ab9f8c155ee1588bf)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:16.378724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:16.558693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7ca1b5ec9e3475ab9f8c155ee1588bf'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:16.566660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7ca1b5ec9e3475ab9f8c155ee1588bf'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:16.567348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7ca1b5ec9e3475ab9f8c155ee1588bf'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:16.567961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7ca1b5ec9e3475ab9f8c155ee1588bf'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:16.618336 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7ca1b5ec9e3475ab9f8c155ee1588bf'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:16.636688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1178/2406] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:16 2022] DELETE /identity/v3/users/e7ca1b5ec9e3475ab9f8c155ee1588bf => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:16.641637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:16.642236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:16.642236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:16.661076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:16.662274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:16.703028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:16.721932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1229/2407] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:16.732642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:16.734071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:16.748211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:16.792392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece8b10a-329c-4ec4-878c-88e30768d16a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:16.796084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:16.796708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:16.796958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:16.797229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:16.806992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:16.808977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:16.819505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1179/2408] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:17.232190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:17.233976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:17.247193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ce4700-2760-40b1-b0aa-890c0afbbd33 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:17.252498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:17.252687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:17.252973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:17.254634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] PATH_INFO: `/v3/projects/c91b5cfec8a943c19bfb5968dea35bea` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:17.254634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c91b5cfec8a943c19bfb5968dea35bea)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:17.256303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:17.381161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c91b5cfec8a943c19bfb5968dea35bea', 'request_id': 'req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.394247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c91b5cfec8a943c19bfb5968dea35bea', 'request_id': 'req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.441717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c91b5cfec8a943c19bfb5968dea35bea', 'request_id': 'req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.452854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] The token cache is being invalidate because project c91b5cfec8a943c19bfb5968dea35bea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:17.453145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.453766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.456080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1230/2409] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:17 2022] DELETE /identity/v3/projects/c91b5cfec8a943c19bfb5968dea35bea => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:17.460068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:17.469644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93008336-2a6e-493e-bf60-0a2a456d9b87 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:17.526771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:17.527407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:17.527685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:17.527998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] PATH_INFO: `/v3/users/0659b0d0222b4f7d8578f6c7665d2d0a` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:17.529060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0659b0d0222b4f7d8578f6c7665d2d0a)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:17.531028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:17.690206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0659b0d0222b4f7d8578f6c7665d2d0a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.697109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0659b0d0222b4f7d8578f6c7665d2d0a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.697772 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0659b0d0222b4f7d8578f6c7665d2d0a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.698435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0659b0d0222b4f7d8578f6c7665d2d0a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.755915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-64bfc82a-a47b-4ac2-8f73-f01701cab737 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0659b0d0222b4f7d8578f6c7665d2d0a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:17.775458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1180/2410] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:17 2022] DELETE /identity/v3/users/0659b0d0222b4f7d8578f6c7665d2d0a => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:18.220279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:18.228198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:18.243335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4253a153-dcb1-4da3-826d-e8fb4e1e9f93 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:18.301827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:18.302485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.302890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.303085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] PATH_INFO: `/v3/projects/895f1b36689342429cc4f60730cd121e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.304189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=895f1b36689342429cc4f60730cd121e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:18.306116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:18.391272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '895f1b36689342429cc4f60730cd121e', 'request_id': 'req-0680f703-aa6f-4557-a19a-4ffcaddabbf8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:18.398142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '895f1b36689342429cc4f60730cd121e', 'request_id': 'req-0680f703-aa6f-4557-a19a-4ffcaddabbf8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:18.452985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '895f1b36689342429cc4f60730cd121e', 'request_id': 'req-0680f703-aa6f-4557-a19a-4ffcaddabbf8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:18.459744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:18.464351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] The token cache is being invalidate because project 895f1b36689342429cc4f60730cd121e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:18.464584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:18.465212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0680f703-aa6f-4557-a19a-4ffcaddabbf8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:18.466787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1231/2411] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:18 2022] DELETE /identity/v3/projects/895f1b36689342429cc4f60730cd121e => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:18.472141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4727bf-b3cb-4d94-9677-958440357a7e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.472644 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4727bf-b3cb-4d94-9677-958440357a7e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.472747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4727bf-b3cb-4d94-9677-958440357a7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.492611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-2e4727bf-b3cb-4d94-9677-958440357a7e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:18.493925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2e4727bf-b3cb-4d94-9677-958440357a7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:18.524348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:18.556398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1181/2412] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:18.564481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978ebf55-8229-4eb8-822b-0e2474f70f07 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.569242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978ebf55-8229-4eb8-822b-0e2474f70f07 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.569242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978ebf55-8229-4eb8-822b-0e2474f70f07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.585071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-978ebf55-8229-4eb8-822b-0e2474f70f07 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:18.586270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-978ebf55-8229-4eb8-822b-0e2474f70f07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:18.594272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1232/2413] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:18.599476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4727bf-b3cb-4d94-9677-958440357a7e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:18.608961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4727bf-b3cb-4d94-9677-958440357a7e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:18.615562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:18.615562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.615562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.615562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.615957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:59:18.616260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:18.617887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:18.629820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1182/2414] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:59:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:59:18.634114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978ebf55-8229-4eb8-822b-0e2474f70f07 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:18.643752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978ebf55-8229-4eb8-822b-0e2474f70f07 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:18.648409 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:18.649008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.649287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.649556 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.650175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:18.652141 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:18.734770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1233/2415] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:18 2022] POST /identity/v3/projects => generated 381 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:18.739775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:18.750082 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fde2869-420c-4843-b1a1-9104213acaae admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:18.809355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:18.809985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.810262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.810608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.811244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:18.813162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:18.916628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1183/2416] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:18 2022] POST /identity/v3/users => generated 291 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:18.920701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:18.940148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c005c99-dc68-496e-8067-e793f8093905 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:18.944896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:18.945506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.945779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.946047 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.946715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:18.948329 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:18.959622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1234/2417] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:18 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:18.963383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:18.974607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663be99-5d13-44e7-be05-95579f43830d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:18.979465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8602c574-b930-4304-8763-a32e4afe89fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:18.980213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602c574-b930-4304-8763-a32e4afe89fe admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:18.980504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602c574-b930-4304-8763-a32e4afe89fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:18.980782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602c574-b930-4304-8763-a32e4afe89fe admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:18.981432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8602c574-b930-4304-8763-a32e4afe89fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:18.983091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8602c574-b930-4304-8763-a32e4afe89fe admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:18.994961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1184/2418] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:18.998649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:19.011482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a047a613-4240-4473-a88f-dc3d7828a61e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:19.016322 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:19.016968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.017345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.017698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] PATH_INFO: `/v3/projects/67ff384fbcc140e7911ea57009a7e791/users/5758f6f1428e43f79f8c1af77c704d64/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.033667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67ff384fbcc140e7911ea57009a7e791, user_id=5758f6f1428e43f79f8c1af77c704d64, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:19.036317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:19.116406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1235/2419] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:18 2022] PUT /identity/v3/projects/67ff384fbcc140e7911ea57009a7e791/users/5758f6f1428e43f79f8c1af77c704d64/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:19.128613 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46f0557-e277-45ee-b354-0f11f8e5d62b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.128937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46f0557-e277-45ee-b354-0f11f8e5d62b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.129200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46f0557-e277-45ee-b354-0f11f8e5d62b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.159013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e46f0557-e277-45ee-b354-0f11f8e5d62b None None] MFA Rules not processed for user `5758f6f1428e43f79f8c1af77c704d64`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:19.160439 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e46f0557-e277-45ee-b354-0f11f8e5d62b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:19.344877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1185/2420] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:19 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:19.349778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:19.412147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012b5dcb-c1cc-4e05-b69e-7e338bfedf03 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:19.416763 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:19.417369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.417650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.417920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.418592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:19.420502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:19.539636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1236/2421] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:19 2022] POST /identity/v3/projects => generated 381 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:19.543705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46f0557-e277-45ee-b354-0f11f8e5d62b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:19.554832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46f0557-e277-45ee-b354-0f11f8e5d62b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:19.579877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:19.612955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:19.617405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.617405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.617405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.617405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:19.617405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:19.691524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1186/2422] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:19 2022] POST /identity/v3/users => generated 297 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:19.692976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:19.703709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ca062-cdab-4c63-ac07-7666ea66925e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:19.708649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:19.709890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.710171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.710469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.711119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:19.712729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:19.724997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1237/2423] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:19.729074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:19.740065 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63f227c-c29b-475a-ad94-3620ffb08f93 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:19.744711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:19.745361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.745636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.746000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] PATH_INFO: `/v3/projects/2d0b83f7ecd8441b8568706f729427f4/users/7e2246ebb8004e8281f872f72257a6f8/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.762160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d0b83f7ecd8441b8568706f729427f4, user_id=7e2246ebb8004e8281f872f72257a6f8, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:19.765138 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:19.830783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1187/2424] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:19 2022] PUT /identity/v3/projects/2d0b83f7ecd8441b8568706f729427f4/users/7e2246ebb8004e8281f872f72257a6f8/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:19.834589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:19.845344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983672aa-89dc-40e9-a79b-24f31ef66262 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:19.909144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c02ea6-2d4d-43ba-be70-4da1345db7b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:19.909669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c02ea6-2d4d-43ba-be70-4da1345db7b9 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.909946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c02ea6-2d4d-43ba-be70-4da1345db7b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.910217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c02ea6-2d4d-43ba-be70-4da1345db7b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.910893 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c02ea6-2d4d-43ba-be70-4da1345db7b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:19.912529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c02ea6-2d4d-43ba-be70-4da1345db7b9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:19.922124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1238/2425] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:19 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:19.927477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:19.942925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645943d-1897-4533-ab17-b287bd0c5a4e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:19.947442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:19.948141 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.948415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.948681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] PATH_INFO: `/v3/projects/2d0b83f7ecd8441b8568706f729427f4/users/7e2246ebb8004e8281f872f72257a6f8/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.950610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d0b83f7ecd8441b8568706f729427f4, user_id=7e2246ebb8004e8281f872f72257a6f8, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:19.953200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:19.963474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1188/2426] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:19 2022] PUT /identity/v3/projects/2d0b83f7ecd8441b8568706f729427f4/users/7e2246ebb8004e8281f872f72257a6f8/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:19.974673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966356c-6708-49ee-a167-05f4fcd943a7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:19.974956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966356c-6708-49ee-a167-05f4fcd943a7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:19.975225 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966356c-6708-49ee-a167-05f4fcd943a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:19.983482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:20.003844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6966356c-6708-49ee-a167-05f4fcd943a7 None None] MFA Rules not processed for user `7e2246ebb8004e8281f872f72257a6f8`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:20.005091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6966356c-6708-49ee-a167-05f4fcd943a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:20.076928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1239/2427] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:19 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:20.148055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:20.149905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:20.210790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08179d09-d800-48e6-8eda-1bb18f7d4761 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:20.214645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:20.215415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:20.216020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:20.216020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:20.225812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:20.227358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:20.239014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1189/2428] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:20 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:20.347653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6966356c-6708-49ee-a167-05f4fcd943a7 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:20.358991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6966356c-6708-49ee-a167-05f4fcd943a7 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:20.363609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9907efb2-94e8-423a-8145-85efec02c7ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:20.365648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907efb2-94e8-423a-8145-85efec02c7ef service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:20.365648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907efb2-94e8-423a-8145-85efec02c7ef service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:20.365648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9907efb2-94e8-423a-8145-85efec02c7ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:20.427724 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907efb2-94e8-423a-8145-85efec02c7ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:20.429209 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9907efb2-94e8-423a-8145-85efec02c7ef service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:20.438197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1240/2429] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:20 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:59:23.530393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:23.589843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cd01a-7173-4456-90d3-d9ee4b48191b service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:23.594293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:23.594941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:23.595215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:23.595483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] PATH_INFO: `/v3/users/5758f6f1428e43f79f8c1af77c704d64` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:23.597205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5758f6f1428e43f79f8c1af77c704d64)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:23.599296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:23.716468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:23.786343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5758f6f1428e43f79f8c1af77c704d64'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:23.794009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5758f6f1428e43f79f8c1af77c704d64'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:23.796068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5758f6f1428e43f79f8c1af77c704d64'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:23.796068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5758f6f1428e43f79f8c1af77c704d64'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:23.847157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5758f6f1428e43f79f8c1af77c704d64'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:23.864592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1190/2430] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:23 2022] DELETE /identity/v3/users/5758f6f1428e43f79f8c1af77c704d64 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:23.869140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:23.869608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:23.869702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:23.889637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:23.890881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:23.952178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1241/2431] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:23.962500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:23.964017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:24.020671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a17d4c-f866-4402-b47c-7ae896e2935a admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:24.024281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:24.024888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:24.025164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:24.025433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:24.034353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:24.035796 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:24.045047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1191/2432] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:24.443701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:24.445421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:24.459935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b900454-feb9-4f79-94d4-8d4d1ec903b8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:24.465335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:24.465977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:24.466259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:24.466568 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] PATH_INFO: `/v3/projects/67ff384fbcc140e7911ea57009a7e791` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:24.467786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=67ff384fbcc140e7911ea57009a7e791)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:24.469780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:24.495673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:24.573358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67ff384fbcc140e7911ea57009a7e791', 'request_id': 'req-06540400-afbb-422d-a95c-8fc9e6a2a7b8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.579521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67ff384fbcc140e7911ea57009a7e791', 'request_id': 'req-06540400-afbb-422d-a95c-8fc9e6a2a7b8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.658642 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67ff384fbcc140e7911ea57009a7e791', 'request_id': 'req-06540400-afbb-422d-a95c-8fc9e6a2a7b8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.670385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] The token cache is being invalidate because project 67ff384fbcc140e7911ea57009a7e791 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:24.670694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.671339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.672823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1242/2433] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:24 2022] DELETE /identity/v3/projects/67ff384fbcc140e7911ea57009a7e791 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:24.677714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:24.687287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0851191c-17d2-4bd1-b06c-76e71d2e487c service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:24.741941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:24.742594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:24.742868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:24.743140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] PATH_INFO: `/v3/users/7e2246ebb8004e8281f872f72257a6f8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:24.744235 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e2246ebb8004e8281f872f72257a6f8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:24.746118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:24.912321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e2246ebb8004e8281f872f72257a6f8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.913472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:24.924515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e2246ebb8004e8281f872f72257a6f8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.925338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e2246ebb8004e8281f872f72257a6f8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.926160 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e2246ebb8004e8281f872f72257a6f8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.973088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-860053bc-4c25-4945-8f12-91e88201b2a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e2246ebb8004e8281f872f72257a6f8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:24.991439 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1192/2434] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:24 2022] DELETE /identity/v3/users/7e2246ebb8004e8281f872f72257a6f8 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:25.426259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:25.427855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:25.442169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06540400-afbb-422d-a95c-8fc9e6a2a7b8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:25.499574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:25.500204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:25.500482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:25.500750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] PATH_INFO: `/v3/projects/2d0b83f7ecd8441b8568706f729427f4` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:25.501845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d0b83f7ecd8441b8568706f729427f4)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:25.504411 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:25.638379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d0b83f7ecd8441b8568706f729427f4', 'request_id': 'req-61756cef-a38a-495f-8572-5e4cd5cd4739'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:25.644892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d0b83f7ecd8441b8568706f729427f4', 'request_id': 'req-61756cef-a38a-495f-8572-5e4cd5cd4739'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:25.709528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d0b83f7ecd8441b8568706f729427f4', 'request_id': 'req-61756cef-a38a-495f-8572-5e4cd5cd4739'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:25.720426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] The token cache is being invalidate because project 2d0b83f7ecd8441b8568706f729427f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:25.720715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:25.721347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-61756cef-a38a-495f-8572-5e4cd5cd4739 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:25.723159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1243/2435] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:25 2022] DELETE /identity/v3/projects/2d0b83f7ecd8441b8568706f729427f4 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:25.728095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4393ab56-69d2-4aec-ac67-640b3deb822e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:25.728563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4393ab56-69d2-4aec-ac67-640b3deb822e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:25.728657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4393ab56-69d2-4aec-ac67-640b3deb822e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:25.751014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-4393ab56-69d2-4aec-ac67-640b3deb822e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:25.767150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4393ab56-69d2-4aec-ac67-640b3deb822e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:25.810672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1193/2436] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:25.819590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56f8594-dbfb-4d66-8144-352fe3aecdca None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:25.819877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56f8594-dbfb-4d66-8144-352fe3aecdca None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:25.820153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56f8594-dbfb-4d66-8144-352fe3aecdca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:25.840877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e56f8594-dbfb-4d66-8144-352fe3aecdca None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:25.842166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e56f8594-dbfb-4d66-8144-352fe3aecdca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:25.850233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1244/2437] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:25.854533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4393ab56-69d2-4aec-ac67-640b3deb822e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:25.864883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4393ab56-69d2-4aec-ac67-640b3deb822e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:25.869315 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:25.869898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:25.870175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:25.870467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:25.871101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:59:25.871396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:25.872961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:25.884860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1194/2438] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:59:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:59:25.889216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56f8594-dbfb-4d66-8144-352fe3aecdca None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:25.898644 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56f8594-dbfb-4d66-8144-352fe3aecdca None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:25.903597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:25.904199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:25.904471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:25.904735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:25.905356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:25.907285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:25.997836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1245/2439] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:25 2022] POST /identity/v3/projects => generated 373 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:26.001847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:26.012044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f8c80-8960-4b8d-9f5f-ba8bd5dbf7b1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:26.068343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:26.068949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:26.069227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:26.069494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:26.070127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:26.072047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:26.127619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1195/2440] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:26 2022] POST /identity/v3/users => generated 287 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:26.131755 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:26.142345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43964501-a42f-4e59-9d90-3e7aab85a6c7 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:26.147298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:26.147935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:26.148212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:26.148487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:26.149235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:26.150775 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:26.161844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1246/2441] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:26.165955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:26.176358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9719e0-200c-4ef1-8228-cb1f957910b2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:26.180980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06e4c83-d8de-4ddb-aac3-353c3bb1645c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:26.181574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e4c83-d8de-4ddb-aac3-353c3bb1645c admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:26.181848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e4c83-d8de-4ddb-aac3-353c3bb1645c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:26.182114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06e4c83-d8de-4ddb-aac3-353c3bb1645c admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:26.182784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06e4c83-d8de-4ddb-aac3-353c3bb1645c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:26.184359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06e4c83-d8de-4ddb-aac3-353c3bb1645c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:26.194911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1196/2442] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:26.198973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:26.204191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:26.209806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e13a81c-5bdb-4d93-9a9c-18e4b234bbff admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:26.214515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:26.215171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:26.215451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:26.215728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] PATH_INFO: `/v3/projects/f9e5915b40b84a859f5ab0fb5892c973/users/6027a9e1cfe74099a92eb3b39ce6361b/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:26.233716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9e5915b40b84a859f5ab0fb5892c973, user_id=6027a9e1cfe74099a92eb3b39ce6361b, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:26.236442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:26.344366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1247/2443] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:26 2022] PUT /identity/v3/projects/f9e5915b40b84a859f5ab0fb5892c973/users/6027a9e1cfe74099a92eb3b39ce6361b/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:26.349895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e599f130-c5a6-478f-a3bf-5fc2ed8bec33 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:26.351615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e599f130-c5a6-478f-a3bf-5fc2ed8bec33 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:26.351615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e599f130-c5a6-478f-a3bf-5fc2ed8bec33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:26.369337 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e599f130-c5a6-478f-a3bf-5fc2ed8bec33 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:26.370585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e599f130-c5a6-478f-a3bf-5fc2ed8bec33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:26.400386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:26.436295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1197/2444] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:59:26.446085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:26.451171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:26.511139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d50f12-ce01-4812-a4fe-b4f3bc04aa54 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:26.515101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:26.515820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:26.516098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:26.516365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:26.526158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:26.527621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:26.537950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1248/2445] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:27.806297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c89dad-7f5c-4652-8953-fd662c746079 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:27.806925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c89dad-7f5c-4652-8953-fd662c746079 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:27.806925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c89dad-7f5c-4652-8953-fd662c746079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:27.839996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-22c89dad-7f5c-4652-8953-fd662c746079 None None] MFA Rules not processed for user `6027a9e1cfe74099a92eb3b39ce6361b`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:27.851144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-22c89dad-7f5c-4652-8953-fd662c746079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:27.911976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1198/2446] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:27.929840 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:27.931314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:27.992396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37079cbf-0a3b-490b-b7e3-29cae64228ef service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:27.996118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:27.996750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:27.997028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:27.997298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:28.005664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:28.007115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:28.016243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1249/2447] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:29.700087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31092f67-589f-4f37-ae2c-f222bb8d6873 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:29.700962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31092f67-589f-4f37-ae2c-f222bb8d6873 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:29.701402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31092f67-589f-4f37-ae2c-f222bb8d6873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:29.721380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-31092f67-589f-4f37-ae2c-f222bb8d6873 None None] MFA Rules not processed for user `6027a9e1cfe74099a92eb3b39ce6361b`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:29.722612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-31092f67-589f-4f37-ae2c-f222bb8d6873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:29.729588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1199/2448] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:38.011959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=61,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:39.715896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=48 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:43.535061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:43.536533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:43.548375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d942dc-4c75-4b69-adef-53d333d3b1c9 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:43.552683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:43.553396 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:43.553676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:43.553947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:43.555713 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:43.568540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:43.570162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:43.580558 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1250/2449] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:43.661110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31092f67-589f-4f37-ae2c-f222bb8d6873 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:43.670941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31092f67-589f-4f37-ae2c-f222bb8d6873 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:43.674703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:43.675306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:43.675582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:43.675847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:43.678172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:43.690181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:43.691642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:43.702608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1200/2450] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:43.815874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:43.826328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537b6a9c-462d-42cf-8d35-655e93338e20 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:43.830366 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:43.831118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:43.831461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:43.831802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:43.833818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:43.846533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:43.848169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:43.859727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1251/2451] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:49.559012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:49.564020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:49.674205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d392a873-eab1-404a-ae0e-aea30e4b2eb0 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:49.680355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:49.681045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:49.681321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:49.681587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:49.683424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:49.699087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:49.700960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:49.721384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1201/2452] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:49.733760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:49.735244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:49.751195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696379c7-f68d-4ac9-ae93-4de68bf2d37d service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:49.763704 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:49.769406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:49.769406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:49.769406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] PATH_INFO: `/v3/users/14d56042415540a3b3ed06578b712ca5` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:49.769406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] RBAC: Authorizing `identity:delete_user(user_id=14d56042415540a3b3ed06578b712ca5)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:49.774875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:49.944005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14d56042415540a3b3ed06578b712ca5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:49.975438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14d56042415540a3b3ed06578b712ca5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:49.976466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14d56042415540a3b3ed06578b712ca5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:49.977274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14d56042415540a3b3ed06578b712ca5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:50.053670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14d56042415540a3b3ed06578b712ca5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:50.093596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1252/2453] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:49 2022] DELETE /identity/v3/users/14d56042415540a3b3ed06578b712ca5 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:50.706636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:50.726238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8db4b7-1ed4-4763-9f18-90cfe1026cd6 service placement] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:50.806623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:50.806623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:50.806623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:50.807726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] PATH_INFO: `/v3/projects/8760be982f804a9aa319b989aa1c3118` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:50.809387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8760be982f804a9aa319b989aa1c3118)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:50.811611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:50.934287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8760be982f804a9aa319b989aa1c3118', 'request_id': 'req-1aa7940f-00ab-4ebc-a939-05d971534e6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:50.985477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:50.987106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:50.996491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8760be982f804a9aa319b989aa1c3118', 'request_id': 'req-1aa7940f-00ab-4ebc-a939-05d971534e6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.006699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8760be982f804a9aa319b989aa1c3118', 'request_id': 'req-1aa7940f-00ab-4ebc-a939-05d971534e6f'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.012550 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b60cfc-f0ac-4326-85b8-64a2c73a1ffb admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:51.040483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] The token cache is being invalidate because project 8760be982f804a9aa319b989aa1c3118 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:51.041040 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.042266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-1aa7940f-00ab-4ebc-a939-05d971534e6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.044313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1202/2454] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:50 2022] DELETE /identity/v3/projects/8760be982f804a9aa319b989aa1c3118 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:51.052803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db55261f-79c6-4657-aeca-d4944eea6aba None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.053177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db55261f-79c6-4657-aeca-d4944eea6aba None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.053680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db55261f-79c6-4657-aeca-d4944eea6aba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.075038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:51.088421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-db55261f-79c6-4657-aeca-d4944eea6aba None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:51.091076 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-db55261f-79c6-4657-aeca-d4944eea6aba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:51.134079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:51.134756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.135034 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.135304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] PATH_INFO: `/v3/users/cb93504f2605489eaf50cf10930b3e46` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.136468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb93504f2605489eaf50cf10930b3e46)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:51.138875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:51.158034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1203/2455] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:51.165660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.165919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.166188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.188128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:51.188128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:51.197873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1204/2456] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:59:51.198668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:51.199925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:51.212317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fd8868-f02c-43e3-9d01-b56f1cf0eaf0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:51.217155 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:51.217768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.218044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.218312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] PATH_INFO: `/v3/users/835a297b6e75416ea9bd1a82b466b2a9` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.219464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] RBAC: Authorizing `identity:delete_user(user_id=835a297b6e75416ea9bd1a82b466b2a9)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:51.221346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:51.324325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb93504f2605489eaf50cf10930b3e46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.325391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:51.331732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb93504f2605489eaf50cf10930b3e46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.335159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb93504f2605489eaf50cf10930b3e46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.335159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb93504f2605489eaf50cf10930b3e46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.406463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb93504f2605489eaf50cf10930b3e46'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.421501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '835a297b6e75416ea9bd1a82b466b2a9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.424959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1253/2457] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:50 2022] DELETE /identity/v3/users/cb93504f2605489eaf50cf10930b3e46 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:51.425766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:51.427672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:51.430714 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '835a297b6e75416ea9bd1a82b466b2a9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.431422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '835a297b6e75416ea9bd1a82b466b2a9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.432053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '835a297b6e75416ea9bd1a82b466b2a9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.496640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b800ae3-7449-4ecc-befa-e47fdafcfae1 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:51.501347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:51.501980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.502255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.502593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.503238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:59:51.503533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:51.505181 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:51.510255 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7c5a40e1-29a1-457a-9077-7c7fbf6f3592 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '835a297b6e75416ea9bd1a82b466b2a9'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.516862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1254/2458] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:59:51.521567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:51.523639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:51.537319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d05203a-d508-4d34-9cf9-ce162d38c074 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:51.542986 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1205/2459] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:51 2022] DELETE /identity/v3/users/835a297b6e75416ea9bd1a82b466b2a9 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:51.607431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:51.608046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.608432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.608609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.609202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:51.611135 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:51.648505 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803cd9-0509-4c90-9a30-4a8cc806d38e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.648887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803cd9-0509-4c90-9a30-4a8cc806d38e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.649194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81803cd9-0509-4c90-9a30-4a8cc806d38e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.673616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-81803cd9-0509-4c90-9a30-4a8cc806d38e None None] MFA Rules not processed for user `6027a9e1cfe74099a92eb3b39ce6361b`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:51.674989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-81803cd9-0509-4c90-9a30-4a8cc806d38e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:51.698061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1255/2460] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:51 2022] POST /identity/v3/projects => generated 385 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:51.699509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:51.702398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:51.740048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1206/2461] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:59:51.743614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81803cd9-0509-4c90-9a30-4a8cc806d38e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:51.765146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81803cd9-0509-4c90-9a30-4a8cc806d38e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:51.784565 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d16009-27a5-4376-9c52-203eb7a87a85 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:51.790389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:51.791316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:51.791468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.791619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.792001 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] PATH_INFO: `/v3/projects/bc8433b42fb74e8bbfef92acedf01259` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.793468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc8433b42fb74e8bbfef92acedf01259)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:51.795871 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:51.847273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:51.848396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:51.848678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:51.849831 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:51.850526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:51.853144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:51.922958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:51.934254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc8433b42fb74e8bbfef92acedf01259', 'request_id': 'req-ec7f741b-d724-4896-8700-bef1997a74f1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.941075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc8433b42fb74e8bbfef92acedf01259', 'request_id': 'req-ec7f741b-d724-4896-8700-bef1997a74f1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:51.968757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1207/2462] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:51 2022] POST /identity/v3/users => generated 293 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:51.973745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:51.986038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767a139-7a76-45e5-9c7f-f3a2e480f3b9 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.012747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc8433b42fb74e8bbfef92acedf01259', 'request_id': 'req-ec7f741b-d724-4896-8700-bef1997a74f1'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.025620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] The token cache is being invalidate because project bc8433b42fb74e8bbfef92acedf01259 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:52.025867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.026692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-ec7f741b-d724-4896-8700-bef1997a74f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.028390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1256/2463] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:51 2022] DELETE /identity/v3/projects/bc8433b42fb74e8bbfef92acedf01259 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:52.034563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258e72d4-a089-45a5-ab5e-194b66a130d8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.034922 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258e72d4-a089-45a5-ab5e-194b66a130d8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.035408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258e72d4-a089-45a5-ab5e-194b66a130d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.050563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.051187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.051648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.052051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.053026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.054694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.057539 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-258e72d4-a089-45a5-ab5e-194b66a130d8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:52.058949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-258e72d4-a089-45a5-ab5e-194b66a130d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.068992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1208/2464] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:52.069168 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1257/2465] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:52.070297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.072387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.075553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befece2-0a48-4837-bc24-ad1b33135fff None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.075897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befece2-0a48-4837-bc24-ad1b33135fff None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.076449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befece2-0a48-4837-bc24-ad1b33135fff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.088125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6b80cf-16b7-4a28-8d7e-23bc588a4722 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.094063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.094711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.094993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.095268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.095921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.097626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.105950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-3befece2-0a48-4837-bc24-ad1b33135fff None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:52.107205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3befece2-0a48-4837-bc24-ad1b33135fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.112566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1209/2466] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:59:52.113539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.117586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1258/2467] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:52.121891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3befece2-0a48-4837-bc24-ad1b33135fff None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.123963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2958c90-ce72-4bd6-8de3-cafcbaaa1ad8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.128952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.129614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.129893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.130712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] PATH_INFO: `/v3/projects/f8320aa7c44c44c59ca0337aa84c2f2d/users/9f175b352f4240b5923331008dd49cd7/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.132203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3befece2-0a48-4837-bc24-ad1b33135fff None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.137220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.137826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.138362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.138661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.139429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:59:52.139618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.141197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.146615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8320aa7c44c44c59ca0337aa84c2f2d, user_id=9f175b352f4240b5923331008dd49cd7, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.149301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.155571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1259/2468] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:59:52.160016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.170773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1078a4e1-7620-463b-bf3a-149e757cd867 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.175582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e36056a-7965-482e-89c4-03513b6ecc4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.176171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36056a-7965-482e-89c4-03513b6ecc4d admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.176453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36056a-7965-482e-89c4-03513b6ecc4d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.176768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36056a-7965-482e-89c4-03513b6ecc4d admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.177532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e36056a-7965-482e-89c4-03513b6ecc4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.179467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e36056a-7965-482e-89c4-03513b6ecc4d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.231241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1210/2469] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:52 2022] PUT /identity/v3/projects/f8320aa7c44c44c59ca0337aa84c2f2d/users/9f175b352f4240b5923331008dd49cd7/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:52.238823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.238823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.323534 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409b0d74-ddc4-4213-90d8-2b2d72a35dfa admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.331799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.332491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.332769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.335683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] PATH_INFO: `/v3/projects/3f3deef8b1cc4c9193e9357f0ba99e15` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.335683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f3deef8b1cc4c9193e9357f0ba99e15)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.336999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.339914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1260/2470] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/projects => generated 385 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:52.342335 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b613af-2a65-4b1e-9583-4d8618ae0289 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.342792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b613af-2a65-4b1e-9583-4d8618ae0289 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.343193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b613af-2a65-4b1e-9583-4d8618ae0289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.390549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:52.395052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-59b613af-2a65-4b1e-9583-4d8618ae0289 None None] MFA Rules not processed for user `9f175b352f4240b5923331008dd49cd7`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:52.396863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-59b613af-2a65-4b1e-9583-4d8618ae0289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.453368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f3deef8b1cc4c9193e9357f0ba99e15', 'request_id': 'req-f53817dd-ba3f-4402-a660-a85953bb77e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.463485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1261/2471] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:52.467948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b613af-2a65-4b1e-9583-4d8618ae0289 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.525498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b613af-2a65-4b1e-9583-4d8618ae0289 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.533024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f3deef8b1cc4c9193e9357f0ba99e15', 'request_id': 'req-f53817dd-ba3f-4402-a660-a85953bb77e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.540711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f3deef8b1cc4c9193e9357f0ba99e15', 'request_id': 'req-f53817dd-ba3f-4402-a660-a85953bb77e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.551184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:52.551637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] The token cache is being invalidate because project 3f3deef8b1cc4c9193e9357f0ba99e15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:52.551923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.552860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:52.554362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1211/2472] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:52 2022] DELETE /identity/v3/projects/3f3deef8b1cc4c9193e9357f0ba99e15 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:52.555140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.556450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.612624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dc63b6-9fa0-46b5-a5df-a5d5c8629762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.613510 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc63b6-9fa0-46b5-a5df-a5d5c8629762 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.614669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc63b6-9fa0-46b5-a5df-a5d5c8629762 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.614669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc63b6-9fa0-46b5-a5df-a5d5c8629762 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.615639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc63b6-9fa0-46b5-a5df-a5d5c8629762 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.617638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25dc63b6-9fa0-46b5-a5df-a5d5c8629762 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.657479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53817dd-ba3f-4402-a660-a85953bb77e8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.664020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.664640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.666250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.666250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.667163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.684591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1262/2473] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/users => generated 293 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:52.686830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.688745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.689463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.715443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:52.716340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.725684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1263/2474] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:52.726272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.727583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.739767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50faea-73e6-40a2-94a1-7a981f901cc5 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.745529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a49d2-c78f-4be5-8631-083aad0aa8b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.746138 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a49d2-c78f-4be5-8631-083aad0aa8b4 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.746440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a49d2-c78f-4be5-8631-083aad0aa8b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.746735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a49d2-c78f-4be5-8631-083aad0aa8b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.747376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5a49d2-c78f-4be5-8631-083aad0aa8b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.748984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5a49d2-c78f-4be5-8631-083aad0aa8b4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.750935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.752524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.764470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1264/2475] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:52.764470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f654af-abcd-4d2b-96ca-81ff1ae05f61 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.764470 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f654af-abcd-4d2b-96ca-81ff1ae05f61 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.764779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f654af-abcd-4d2b-96ca-81ff1ae05f61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.766518 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1212/2476] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:52.767275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.778656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a90e5-b779-4a0e-9123-3369d12eaaf6 service nova] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.783545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.784257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.784645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.785009 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.785745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.788064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.794314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-66f654af-abcd-4d2b-96ca-81ff1ae05f61 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:52.795684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-66f654af-abcd-4d2b-96ca-81ff1ae05f61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.802551 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1213/2477] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:52.803721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.810256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1265/2478] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:52.811048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f654af-abcd-4d2b-96ca-81ff1ae05f61 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.813777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7917cd82-6329-4e9b-8d36-a9301734228e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.820614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d468ed-d0ef-4ad4-949c-401d411790d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.821498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d468ed-d0ef-4ad4-949c-401d411790d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.821851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d468ed-d0ef-4ad4-949c-401d411790d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.822232 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d468ed-d0ef-4ad4-949c-401d411790d9 admin admin] PATH_INFO: `/v3/projects/2d1fd8f799ea4fbbafc99fd6d38b0c7f/users/a49462d299994eb4aa193522d0eeb996/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.823573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f654af-abcd-4d2b-96ca-81ff1ae05f61 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.829502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.830468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.830716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.834345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.834345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:59:52.834345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.834837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.840819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d468ed-d0ef-4ad4-949c-401d411790d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d1fd8f799ea4fbbafc99fd6d38b0c7f, user_id=a49462d299994eb4aa193522d0eeb996, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.843640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d468ed-d0ef-4ad4-949c-401d411790d9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.848537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1266/2479] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:59:52.853120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:52.863494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6661b9b-07e1-4888-b6e7-842540ffa60d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:52.868665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:52.869268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.869542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.869813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.870456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:52.872343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:52.917494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:52.922442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1214/2480] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:52 2022] PUT /identity/v3/projects/2d1fd8f799ea4fbbafc99fd6d38b0c7f/users/a49462d299994eb4aa193522d0eeb996/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:52.933461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:52.933737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:52.934006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:52.962653 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] MFA Rules not processed for user `a49462d299994eb4aa193522d0eeb996`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:52.963854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:52.989136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1267/2481] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/projects => generated 355 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:52.992433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.040731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1215/2482] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:59:53.048223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.050022 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:53.057685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:53.074009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721a4b1d-9178-42bd-a2bb-7abc974c3773 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.078876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.079618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.080167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.080405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.081042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.082979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.110786 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da3d639-9b6f-4df7-be08-123628fb83e3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.157581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.158180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.158509 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.158783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.215684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.217149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.227605 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1216/2483] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:59:53.247277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1268/2484] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:52 2022] POST /identity/v3/users => generated 278 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:53.248440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.256598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.262266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992f1422-de0f-498b-83b6-1bf904e576db service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.268119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4837874b-98e5-4311-9032-b8dbb4aed3c7 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.269274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.273275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.273275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.273275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.273577 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.273710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.274125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.274429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.274708 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] PATH_INFO: `/v3/users/a49462d299994eb4aa193522d0eeb996` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.275222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.275812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a49462d299994eb4aa193522d0eeb996)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.277737 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.289041 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1217/2485] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:53 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:53.290334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.300128 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95293ae5-7e97-4063-9abb-24cc0eefd13e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.307347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.307347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.307347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.307347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.307872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.308950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.319166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1218/2486] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:59:53.323070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.332430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c428964b-29f3-40c5-a444-f4d42ff71d60 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.337874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5a7ea-e94c-4bc1-952b-e2dc090b88f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.341841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5a7ea-e94c-4bc1-952b-e2dc090b88f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.341841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5a7ea-e94c-4bc1-952b-e2dc090b88f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.341841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5a7ea-e94c-4bc1-952b-e2dc090b88f5 admin admin] PATH_INFO: `/v3/projects/597b9561720a496496d4793b898811cb/users/ba45f4a0f59a4e688879a0d734e484c4/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.354654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5a7ea-e94c-4bc1-952b-e2dc090b88f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=597b9561720a496496d4793b898811cb, user_id=ba45f4a0f59a4e688879a0d734e484c4, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.357504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5a7ea-e94c-4bc1-952b-e2dc090b88f5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.464163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1219/2487] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:53 2022] PUT /identity/v3/projects/597b9561720a496496d4793b898811cb/users/ba45f4a0f59a4e688879a0d734e484c4/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:53.474496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.474780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.475049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.506722 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a49462d299994eb4aa193522d0eeb996'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:53.515165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a49462d299994eb4aa193522d0eeb996'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:53.516052 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a49462d299994eb4aa193522d0eeb996'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:53.516851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a49462d299994eb4aa193522d0eeb996'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:53.532845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] MFA Rules not processed for user `ba45f4a0f59a4e688879a0d734e484c4`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:53.534086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:53.556728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:53.605478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1220/2488] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:53 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:59:53.609780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.611126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:53.641792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f2a79337-92bb-4bfa-8dba-7b1c10f6c80b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a49462d299994eb4aa193522d0eeb996'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:53.666102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1269/2489] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 08:59:53 2022] DELETE /identity/v3/users/a49462d299994eb4aa193522d0eeb996 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:53.668293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.668681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.669171 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.690436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1962df-f74c-4f0c-864a-ff81a88409fd None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.724508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:53.727406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:53.751280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:53.761647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.762265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.762581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.762853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.763752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.765678 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.820865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1270/2490] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:59:53.830202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.831585 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:53.865967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1221/2491] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:53 2022] POST /identity/v3/projects => generated 355 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:53.870225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:53.891087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921094f5-1c0d-475d-a471-ccbba183635f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.899619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:53.942392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.943237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.943359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.943648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.951120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6404944b-0441-4661-909e-efc4af6edeac admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:53.952674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.954159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.956264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:53.956896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:53.957175 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:53.957443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:53.958092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:53.960737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:53.964450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1271/2492] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:59:54.057499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1222/2493] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:53 2022] POST /identity/v3/users => generated 284 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:54.059124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.069753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073dfde9-980b-4423-b5c7-a66e53294d67 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.074732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.075489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.076042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.076316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.076956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.078586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.089327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1272/2494] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:54.093677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.104806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1382923-4a64-46b7-bd58-b7bc15540951 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.109448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.110107 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.110381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.110683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] PATH_INFO: `/v3/projects/17d76effb63448c0bd12f08b454c1006/users/fb64f0d23a53492b977ae0fc91ff13e8/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.127236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17d76effb63448c0bd12f08b454c1006, user_id=fb64f0d23a53492b977ae0fc91ff13e8, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.129835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.214286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1223/2495] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:54 2022] PUT /identity/v3/projects/17d76effb63448c0bd12f08b454c1006/users/fb64f0d23a53492b977ae0fc91ff13e8/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:54.218320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.229808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b069ba-4758-4b8b-99c1-2af6db764645 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.294693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c2c38-faee-47ac-b190-7348de35b555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.295426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c2c38-faee-47ac-b190-7348de35b555 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.295764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c2c38-faee-47ac-b190-7348de35b555 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.296098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c2c38-faee-47ac-b190-7348de35b555 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.296817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c2c38-faee-47ac-b190-7348de35b555 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.298494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820c2c38-faee-47ac-b190-7348de35b555 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.312588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1273/2496] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:54.313705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.324646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d6798b-6567-4da4-8e9c-3506d5251396 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.329277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.333947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.333947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.333947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] PATH_INFO: `/v3/projects/17d76effb63448c0bd12f08b454c1006/users/fb64f0d23a53492b977ae0fc91ff13e8/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.333947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17d76effb63448c0bd12f08b454c1006, user_id=fb64f0d23a53492b977ae0fc91ff13e8, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.335651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.347355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1224/2497] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:54 2022] PUT /identity/v3/projects/17d76effb63448c0bd12f08b454c1006/users/fb64f0d23a53492b977ae0fc91ff13e8/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:54.359609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.359833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.360294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.363835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.365872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:54.401479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] MFA Rules not processed for user `fb64f0d23a53492b977ae0fc91ff13e8`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:54.408264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:54.444673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85038d3-c3bf-46f8-9d62-0f008fd887a6 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.450807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.451446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.451728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.452003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] PATH_INFO: `/v3/projects/2d1fd8f799ea4fbbafc99fd6d38b0c7f` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.453161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d1fd8f799ea4fbbafc99fd6d38b0c7f)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.455542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.484370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1274/2498] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:54 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:54.499162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.500567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:54.528946 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:54.556924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3945b8a-d125-4906-977f-7d618cec5208 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.560561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f54b05-5718-4f8d-8d08-91f21b377508 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.561163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f54b05-5718-4f8d-8d08-91f21b377508 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.561440 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f54b05-5718-4f8d-8d08-91f21b377508 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.561710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f54b05-5718-4f8d-8d08-91f21b377508 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.571085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f54b05-5718-4f8d-8d08-91f21b377508 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.572506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f54b05-5718-4f8d-8d08-91f21b377508 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.581772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1275/2499] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:54 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:59:54.584699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:INSERT=6,keystone:DELETE=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:54.599614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d1fd8f799ea4fbbafc99fd6d38b0c7f', 'request_id': 'req-a015120f-2511-4152-870a-9e3bd3d1269d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:54.704489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d1fd8f799ea4fbbafc99fd6d38b0c7f', 'request_id': 'req-a015120f-2511-4152-870a-9e3bd3d1269d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:54.716456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d1fd8f799ea4fbbafc99fd6d38b0c7f', 'request_id': 'req-a015120f-2511-4152-870a-9e3bd3d1269d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:54.735392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] The token cache is being invalidate because project 2d1fd8f799ea4fbbafc99fd6d38b0c7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:59:54.735392 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:54.736115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a015120f-2511-4152-870a-9e3bd3d1269d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:59:54.743303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1225/2500] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 08:59:54 2022] DELETE /identity/v3/projects/2d1fd8f799ea4fbbafc99fd6d38b0c7f => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:59:54.745622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec926c-c24a-4cca-9d29-f81f6c10c341 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.745972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec926c-c24a-4cca-9d29-f81f6c10c341 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.746178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebec926c-c24a-4cca-9d29-f81f6c10c341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.769219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ebec926c-c24a-4cca-9d29-f81f6c10c341 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:54.770447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ebec926c-c24a-4cca-9d29-f81f6c10c341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:54.832865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1276/2501] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:54.842620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.843199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.843296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.863565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:54.866448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:54.877744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1226/2502] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:59:54.883294 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec926c-c24a-4cca-9d29-f81f6c10c341 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.896010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebec926c-c24a-4cca-9d29-f81f6c10c341 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.902167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.902813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.903091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.903359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.904389 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:59:54.904681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.904799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.906052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:54.906352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.922469 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1277/2503] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 08:59:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:59:54.925288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:54.940070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c5c517-01d7-4732-8566-6846200f8bc4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.951199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.954305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.954682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.955342 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.957874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:54.960339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:54.987465 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a396dd9-7d1c-4c4a-8333-e9616fd1d1f2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:54.992601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:54.993222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:54.993495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:54.993762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:54.995795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:55.072537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:55.074189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:55.086709 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1227/2504] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:54 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:59:55.096356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1278/2505] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 08:59:54 2022] POST /identity/v3/projects => generated 383 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:59:55.100818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:55.174907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183b99a9-388d-4ec2-a918-33a4d81985e7 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:55.191767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:55.244592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:55.245356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:55.245632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:55.245902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:55.246568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:55.248788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:55.372675 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1228/2506] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 08:59:55 2022] POST /identity/v3/users => generated 292 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:59:55.373941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:55.424374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae06f18-3d05-4432-940d-7e944c33d8d7 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:55.430009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:55.430656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:55.430939 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:55.431215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:55.431859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:55.433478 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:55.444276 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1279/2507] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:55 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:55.448028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:55.457872 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930f54b6-e024-4e5e-9753-87fe3ca14394 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:55.463880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d18d77c-261e-4c01-a2e0-534f3fa83a64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:55.477479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d18d77c-261e-4c01-a2e0-534f3fa83a64 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:55.477479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d18d77c-261e-4c01-a2e0-534f3fa83a64 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:55.477479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d18d77c-261e-4c01-a2e0-534f3fa83a64 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:55.477479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d18d77c-261e-4c01-a2e0-534f3fa83a64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:55.477479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d18d77c-261e-4c01-a2e0-534f3fa83a64 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:55.484823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1229/2508] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 08:59:55 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:59:55.488920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:55.501381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d712f8be-4562-417e-ad51-6cc7efc49078 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:55.507251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:55.507957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:55.508236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:55.508507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] PATH_INFO: `/v3/projects/b7535414d7a64f5bb1ec0195abbfb301/users/4e15792706144097b361ed7604a47dd9/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:55.527466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7535414d7a64f5bb1ec0195abbfb301, user_id=4e15792706144097b361ed7604a47dd9, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:55.530272 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:55.607680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1280/2509] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 08:59:55 2022] PUT /identity/v3/projects/b7535414d7a64f5bb1ec0195abbfb301/users/4e15792706144097b361ed7604a47dd9/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:59:55.618635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:55.618919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:55.619191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:55.647341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] MFA Rules not processed for user `4e15792706144097b361ed7604a47dd9`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:59:55.648554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:59:55.718063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1230/2510] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 08:59:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:59:55.729536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:59:55.745129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:59:55.804442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9768d26f-2cbc-4ec0-8434-4fe845cb92d2 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:59:55.808257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d7d97b-6382-4322-9a5a-9c3336d96438 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:59:55.808864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7d97b-6382-4322-9a5a-9c3336d96438 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:59:55.811439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7d97b-6382-4322-9a5a-9c3336d96438 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:59:55.811439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7d97b-6382-4322-9a5a-9c3336d96438 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:59:55.818190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d7d97b-6382-4322-9a5a-9c3336d96438 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:59:55.819776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d7d97b-6382-4322-9a5a-9c3336d96438 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:59:55.828428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1281/2511] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 08:59:55 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:03.325808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:03.325808 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:03.402112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde6ee4d-613e-4ea9-8c35-886c1320e608 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:03.406886 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:03.407519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:03.407778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:03.408051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] PATH_INFO: `/v3/users/9f175b352f4240b5923331008dd49cd7` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:03.409311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f175b352f4240b5923331008dd49cd7)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:03.411938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:03.506884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:03.600018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f175b352f4240b5923331008dd49cd7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:03.608240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f175b352f4240b5923331008dd49cd7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:03.610125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f175b352f4240b5923331008dd49cd7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:03.610773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f175b352f4240b5923331008dd49cd7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:03.686200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f175b352f4240b5923331008dd49cd7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:03.714751 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1231/2512] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:03 2022] DELETE /identity/v3/users/9f175b352f4240b5923331008dd49cd7 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:03.721005 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:03.721495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:03.721673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:03.753870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:03.755754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:03.821401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1282/2513] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:03.831497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:03.832974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:03.894780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d01c775-eafb-4f08-82cb-28c8c67cce94 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:03.899258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4a18be-cc41-4dd9-a1ac-2eeec577b0fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:03.899870 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4a18be-cc41-4dd9-a1ac-2eeec577b0fd service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:03.900148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4a18be-cc41-4dd9-a1ac-2eeec577b0fd service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:03.900412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4a18be-cc41-4dd9-a1ac-2eeec577b0fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:03.909687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4a18be-cc41-4dd9-a1ac-2eeec577b0fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:03.911145 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4a18be-cc41-4dd9-a1ac-2eeec577b0fd service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:03.921912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1232/2514] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:04.466512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:04.467913 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:04.481124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c156a8-620b-4633-9f6e-57ea1a947d10 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:04.490662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:04.492412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:04.492952 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:04.493372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] PATH_INFO: `/v3/projects/f8320aa7c44c44c59ca0337aa84c2f2d` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:04.494750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8320aa7c44c44c59ca0337aa84c2f2d)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:04.497264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:04.639103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8320aa7c44c44c59ca0337aa84c2f2d', 'request_id': 'req-917f014f-355b-4dfe-9990-9639cb0e0e84'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:04.648520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8320aa7c44c44c59ca0337aa84c2f2d', 'request_id': 'req-917f014f-355b-4dfe-9990-9639cb0e0e84'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:04.740669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8320aa7c44c44c59ca0337aa84c2f2d', 'request_id': 'req-917f014f-355b-4dfe-9990-9639cb0e0e84'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:04.760877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] The token cache is being invalidate because project f8320aa7c44c44c59ca0337aa84c2f2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:04.761498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:04.762667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-917f014f-355b-4dfe-9990-9639cb0e0e84 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:04.766079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1283/2515] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:04 2022] DELETE /identity/v3/projects/f8320aa7c44c44c59ca0337aa84c2f2d => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:04.772716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9d9f6-db2e-4988-9f93-7ec652fca27e None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:04.773298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9d9f6-db2e-4988-9f93-7ec652fca27e None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:04.773298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9d9f6-db2e-4988-9f93-7ec652fca27e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:04.874843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a3d9d9f6-db2e-4988-9f93-7ec652fca27e None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:04.874843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a3d9d9f6-db2e-4988-9f93-7ec652fca27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:04.932697 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1233/2516] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:04.940831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9039db67-b31a-4f40-854a-caea71be92d8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:04.941114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9039db67-b31a-4f40-854a-caea71be92d8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:04.941442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9039db67-b31a-4f40-854a-caea71be92d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:04.963044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-9039db67-b31a-4f40-854a-caea71be92d8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:04.966006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-9039db67-b31a-4f40-854a-caea71be92d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:04.972283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1284/2517] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:04.976759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9d9f6-db2e-4988-9f93-7ec652fca27e None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:04.988380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9d9f6-db2e-4988-9f93-7ec652fca27e None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:04.993513 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:04.994711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:04.994711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:04.994930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:04.995419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:00:04.995712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:04.997291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.010354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1234/2518] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:00:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:00:05.014949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9039db67-b31a-4f40-854a-caea71be92d8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.025839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9039db67-b31a-4f40-854a-caea71be92d8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.030571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.031175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.031454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.031729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.032364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.034285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.046685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:05.128947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1285/2519] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:05 2022] POST /identity/v3/projects => generated 369 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:05.133723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.144064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605b4a42-6e23-4d44-b765-42cd762b8d09 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.152681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:05.204959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.205527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.205804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.206070 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.206737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.208705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.300268 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1235/2520] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:05 2022] POST /identity/v3/users => generated 285 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:05.305137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.315319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f22ae07-be21-4ef6-8049-cf264737d87f admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.320726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.321314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.321590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.321864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.322523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.324103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.334068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1286/2521] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:05 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:05.338172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.348578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2200b98-9312-4779-97c0-a5b9b587128f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.353923 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be07e01-9dc8-4779-81e7-9af862bb7392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.354592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be07e01-9dc8-4779-81e7-9af862bb7392 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.354877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be07e01-9dc8-4779-81e7-9af862bb7392 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.355150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be07e01-9dc8-4779-81e7-9af862bb7392 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.355828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be07e01-9dc8-4779-81e7-9af862bb7392 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.357461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be07e01-9dc8-4779-81e7-9af862bb7392 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.368967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1236/2522] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:05.373369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.384086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a432f2-dd0d-44b3-9302-a9aa8c0a797c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.389228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.389887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.390193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.390504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] PATH_INFO: `/v3/projects/13e79bfb82b84cd693ccb7faa3c0c668/users/b2c2db836b2b4a0cab635d2546950b83/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.405650 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=13e79bfb82b84cd693ccb7faa3c0c668, user_id=b2c2db836b2b4a0cab635d2546950b83, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.408331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.493977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1287/2523] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:05 2022] PUT /identity/v3/projects/13e79bfb82b84cd693ccb7faa3c0c668/users/b2c2db836b2b4a0cab635d2546950b83/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:05.504265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9301e38e-dbc9-425f-acd1-97391d3bbfa0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.504627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9301e38e-dbc9-425f-acd1-97391d3bbfa0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.504929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9301e38e-dbc9-425f-acd1-97391d3bbfa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.533900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9301e38e-dbc9-425f-acd1-97391d3bbfa0 None None] MFA Rules not processed for user `b2c2db836b2b4a0cab635d2546950b83`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:05.535196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9301e38e-dbc9-425f-acd1-97391d3bbfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:05.604649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1237/2524] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:05 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:05.609083 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.669532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe99421-eb31-49de-bb36-f6c1797dd6fe admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.674118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.674748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.675026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.675293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.675907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.678604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.768849 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1288/2525] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:05 2022] POST /identity/v3/projects => generated 371 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:05.773116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9301e38e-dbc9-425f-acd1-97391d3bbfa0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.783821 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9301e38e-dbc9-425f-acd1-97391d3bbfa0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.841486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.842105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.842390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.842526 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.842691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.843333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.844196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:05.845231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.858646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eefe8f-076f-4ef3-9e07-8b431997a239 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.863736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.865050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.865422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.865890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] PATH_INFO: `/v3/users/4e15792706144097b361ed7604a47dd9` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.867137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e15792706144097b361ed7604a47dd9)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.869100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.934639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1238/2526] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:05 2022] POST /identity/v3/users => generated 292 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:05.938425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.948303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608513c7-5c80-49d8-84e6-146e048d306b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.953226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.953823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.964690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.964690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:05.964690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:05.964690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:05.964690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:05.968606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1239/2527] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:00:05.973566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:05.984237 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff7984b-642c-4ee3-b2ae-4efb136ad638 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:05.992676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:05.993346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:05.993625 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:05.993897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] PATH_INFO: `/v3/projects/30c05d9f04374159abfe30618575a755/users/77f86bd2face4ef18d1319960271e953/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.011767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30c05d9f04374159abfe30618575a755, user_id=77f86bd2face4ef18d1319960271e953, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:06.012596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e15792706144097b361ed7604a47dd9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:06.015890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:06.020284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e15792706144097b361ed7604a47dd9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:06.021452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e15792706144097b361ed7604a47dd9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:06.022137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e15792706144097b361ed7604a47dd9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:06.093553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-484d9ea0-c199-4492-bda9-56ac887a1c56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e15792706144097b361ed7604a47dd9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:06.104597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1240/2528] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:05 2022] PUT /identity/v3/projects/30c05d9f04374159abfe30618575a755/users/77f86bd2face4ef18d1319960271e953/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:06.113161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:06.115099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.118261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1289/2529] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:05 2022] DELETE /identity/v3/users/4e15792706144097b361ed7604a47dd9 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:06.122850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:06.123139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:06.123414 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.142532 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:06.143742 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.150111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:06.199050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc01b0-4af5-4f2c-ac89-7cec2362dea2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:06.222042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1290/2530] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:06.230219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:06.232183 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.284426 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:06.285111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:06.285388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:06.285695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.287028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:06.288997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:06.301472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1241/2531] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:06 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:00:06.310925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:06.315899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c075ac0-40ff-44a0-97f1-29f66595f795 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:06.321316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b319814-19eb-4b3b-855d-9b144bf0ae4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:06.322052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b319814-19eb-4b3b-855d-9b144bf0ae4a admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:06.322469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b319814-19eb-4b3b-855d-9b144bf0ae4a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:06.322631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d76cd-6cfa-41e6-8a2d-d5e409f6ef77 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:06.322874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b319814-19eb-4b3b-855d-9b144bf0ae4a admin admin] PATH_INFO: `/v3/projects/30c05d9f04374159abfe30618575a755/users/77f86bd2face4ef18d1319960271e953/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.324959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b319814-19eb-4b3b-855d-9b144bf0ae4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=30c05d9f04374159abfe30618575a755, user_id=77f86bd2face4ef18d1319960271e953, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:06.333700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:06.334063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b319814-19eb-4b3b-855d-9b144bf0ae4a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:06.334249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:06.334249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:06.334249 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.341087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1242/2532] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:06 2022] PUT /identity/v3/projects/30c05d9f04374159abfe30618575a755/users/77f86bd2face4ef18d1319960271e953/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:06.344889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:06.346553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:06.353542 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:06.356252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:06.356252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.358046 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1291/2533] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:00:06.387671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] MFA Rules not processed for user `77f86bd2face4ef18d1319960271e953`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:06.391676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.463997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:06.488077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1243/2534] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:06 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:00:06.499735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:06.501224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.560443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ede885-4c2a-4db5-8acd-b0e2d1f5c50d service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:06.612523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:06.613477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:06.613761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:06.614035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.625023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:06.629240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:06.638151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1292/2535] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:06 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:06.909195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:06.911412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.924110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfec0da-f64e-45c2-ac87-9a7a09a8ac0d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:06.928323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5c3c1d-22af-45a6-91a5-7ebf3d5c05af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:06.928942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c3c1d-22af-45a6-91a5-7ebf3d5c05af service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:06.929204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c3c1d-22af-45a6-91a5-7ebf3d5c05af service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:06.929473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c3c1d-22af-45a6-91a5-7ebf3d5c05af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:06.931200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6a5c3c1d-22af-45a6-91a5-7ebf3d5c05af service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.974606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:06.976126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:06.992388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:06.995987 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5c3c1d-22af-45a6-91a5-7ebf3d5c05af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:06.997604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5c3c1d-22af-45a6-91a5-7ebf3d5c05af service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.010776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1244/2536] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:06 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:07.048953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24be9c82-551d-465f-9efa-c343dfff5237 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:07.054639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:07.055589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.055865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.056139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] PATH_INFO: `/v3/projects/b7535414d7a64f5bb1ec0195abbfb301` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.057317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7535414d7a64f5bb1ec0195abbfb301)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:07.059454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.195292 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7535414d7a64f5bb1ec0195abbfb301', 'request_id': 'req-e4d09674-0a05-4143-b21e-8624fdd418fb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:07.201655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7535414d7a64f5bb1ec0195abbfb301', 'request_id': 'req-e4d09674-0a05-4143-b21e-8624fdd418fb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:07.297271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7535414d7a64f5bb1ec0195abbfb301', 'request_id': 'req-e4d09674-0a05-4143-b21e-8624fdd418fb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:07.309368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] The token cache is being invalidate because project b7535414d7a64f5bb1ec0195abbfb301 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:07.309684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:07.310324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4d09674-0a05-4143-b21e-8624fdd418fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:07.311854 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1293/2537] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:06 2022] DELETE /identity/v3/projects/b7535414d7a64f5bb1ec0195abbfb301 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:07.316648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748d6096-84f2-4ab0-9264-9d3018a49b07 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.316941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748d6096-84f2-4ab0-9264-9d3018a49b07 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.317238 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748d6096-84f2-4ab0-9264-9d3018a49b07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.337933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-748d6096-84f2-4ab0-9264-9d3018a49b07 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:07.339192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-748d6096-84f2-4ab0-9264-9d3018a49b07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:07.399822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1245/2538] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:07.407818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d231e8d-ae7c-4ad3-ba11-4aabe3201df8 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.408106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d231e8d-ae7c-4ad3-ba11-4aabe3201df8 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.408380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d231e8d-ae7c-4ad3-ba11-4aabe3201df8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.430754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-5d231e8d-ae7c-4ad3-ba11-4aabe3201df8 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:07.431969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5d231e8d-ae7c-4ad3-ba11-4aabe3201df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:07.440179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1294/2539] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:07.445367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748d6096-84f2-4ab0-9264-9d3018a49b07 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:07.456897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748d6096-84f2-4ab0-9264-9d3018a49b07 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:07.464199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:07.464199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.464199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.464199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.464546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:00:07.464797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:07.466291 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.480099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1246/2540] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:00:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:00:07.484904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d231e8d-ae7c-4ad3-ba11-4aabe3201df8 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:07.495408 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d231e8d-ae7c-4ad3-ba11-4aabe3201df8 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:07.500275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:07.500900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.501174 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.501438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.502112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:07.504178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.568481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1295/2541] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:07 2022] POST /identity/v3/projects => generated 375 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:07.572735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:07.582674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589f5dec-2a80-451a-8fbc-9a6db4f54527 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:07.639105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:07.639711 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.639991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.640263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.640888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:07.642906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.728867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1247/2542] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:07 2022] POST /identity/v3/users => generated 288 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:07.733040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:07.744495 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620c2324-d259-494f-b1a8-aee371b2f7d0 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:07.749467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:07.750080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.750360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.750669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.751320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:07.752933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.764102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1296/2543] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:07.768087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:07.775043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:07.778906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184e1af1-5ba3-4e3b-addb-f5c2611c6fd6 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:07.783748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bc9bed-52b5-48f9-8a2f-949002ebdc43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:07.784342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc9bed-52b5-48f9-8a2f-949002ebdc43 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.784621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc9bed-52b5-48f9-8a2f-949002ebdc43 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.784895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc9bed-52b5-48f9-8a2f-949002ebdc43 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.785531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bc9bed-52b5-48f9-8a2f-949002ebdc43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:07.787866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bc9bed-52b5-48f9-8a2f-949002ebdc43 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.798629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1248/2544] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:07.802900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:07.812897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af01572-5f44-4bba-9266-f3b30e21cbec admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:07.818113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:07.818885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.821597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.821597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] PATH_INFO: `/v3/projects/422aef83dc3a4d0f91707a491f9993f6/users/47c0b4f34d6349cd9fd460996f52b8f8/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.836288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=422aef83dc3a4d0f91707a491f9993f6, user_id=47c0b4f34d6349cd9fd460996f52b8f8, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:07.839102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:07.922756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1297/2545] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:07 2022] PUT /identity/v3/projects/422aef83dc3a4d0f91707a491f9993f6/users/47c0b4f34d6349cd9fd460996f52b8f8/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:07.935490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e44056-6476-4f23-a1af-35e015c72a05 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:07.936019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e44056-6476-4f23-a1af-35e015c72a05 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:07.936019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e44056-6476-4f23-a1af-35e015c72a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:07.964774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a9e44056-6476-4f23-a1af-35e015c72a05 None None] MFA Rules not processed for user `47c0b4f34d6349cd9fd460996f52b8f8`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:07.966000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a9e44056-6476-4f23-a1af-35e015c72a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:08.048885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1249/2546] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:07 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:08.054330 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:08.107144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:08.132049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb1ef4b-380a-40b2-9277-ca86b6c1d6f4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:08.136736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:08.137430 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:08.137712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:08.138103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:08.139111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:08.141167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:08.228405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1298/2547] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:08 2022] POST /identity/v3/projects => generated 377 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:08.229587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e44056-6476-4f23-a1af-35e015c72a05 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:08.240320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e44056-6476-4f23-a1af-35e015c72a05 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:08.297308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:08.297925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:08.298205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:08.298508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:08.299150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:08.301054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:08.422995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1250/2548] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:08 2022] POST /identity/v3/users => generated 289 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:08.427195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:08.437688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec0e547-d6b7-4cc1-b7b6-67a7fad4884a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:08.442325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:08.443166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:08.443441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:08.443736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:08.444393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:08.445972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:08.455576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1299/2549] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:08.459904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:08.470270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8512271b-2aa9-4be3-84c0-a2c3937cf432 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:08.476085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91403b3-fdef-497b-85d4-815d996eb8b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:08.480545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91403b3-fdef-497b-85d4-815d996eb8b5 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:08.480545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91403b3-fdef-497b-85d4-815d996eb8b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:08.480545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91403b3-fdef-497b-85d4-815d996eb8b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:08.480545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91403b3-fdef-497b-85d4-815d996eb8b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:08.480545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91403b3-fdef-497b-85d4-815d996eb8b5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:08.490676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1251/2550] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:08.494926 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:08.505002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1fadde-f967-435e-ad85-10c4bdf6ee95 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:08.516065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:08.516741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:08.517025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:08.517297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] PATH_INFO: `/v3/projects/051583a767f84166ada94c1ea9d5bdac/users/dd908eebceb14ab8801569458b8a9311/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:08.535032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=051583a767f84166ada94c1ea9d5bdac, user_id=dd908eebceb14ab8801569458b8a9311, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:08.538487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:08.624500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1300/2551] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:08 2022] PUT /identity/v3/projects/051583a767f84166ada94c1ea9d5bdac/users/dd908eebceb14ab8801569458b8a9311/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:08.637890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d38dbf-b66c-413f-ae71-c20a62f77ad2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:08.638181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d38dbf-b66c-413f-ae71-c20a62f77ad2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:08.638764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d38dbf-b66c-413f-ae71-c20a62f77ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:08.678768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-06d38dbf-b66c-413f-ae71-c20a62f77ad2 None None] MFA Rules not processed for user `dd908eebceb14ab8801569458b8a9311`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:08.680660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-06d38dbf-b66c-413f-ae71-c20a62f77ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:08.793495 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:08.815660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1252/2552] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:08.820693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:08.902916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f5d7b-8c7e-4a5d-adfc-285c661825f4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:08.907787 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:08.908543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:08.914725 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:08.916819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:08.916819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:08.919136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:08.995629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1301/2553] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:08 2022] POST /identity/v3/projects => generated 377 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:08.996874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d38dbf-b66c-413f-ae71-c20a62f77ad2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:09.008984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d38dbf-b66c-413f-ae71-c20a62f77ad2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:09.068764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:09.069377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:09.069652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:09.069922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:09.070582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:09.072564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:09.168364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1253/2554] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:08 2022] POST /identity/v3/users => generated 295 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:09.171132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:09.183653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbf380-023d-4ca1-9bbe-beddab4a4dcb admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:09.195570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:09.196768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:09.196768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:09.196768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:09.197351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:09.199481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:09.227487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:09.227969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1302/2555] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:09 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:09.240214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de0db58-ec16-4c2c-8714-79208e9a754d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:09.248818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:09.249489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:09.249757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:09.250023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] PATH_INFO: `/v3/projects/7a8dc35c72f44d6f88020e6d44147da8/users/6452014708c04c6aac55c7b92aae554a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:09.272202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a8dc35c72f44d6f88020e6d44147da8, user_id=6452014708c04c6aac55c7b92aae554a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:09.276627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:09.401926 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1254/2556] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:09 2022] PUT /identity/v3/projects/7a8dc35c72f44d6f88020e6d44147da8/users/6452014708c04c6aac55c7b92aae554a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:09.405841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:09.416773 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3364927-54e2-4477-885b-0b3ec8bd85ff admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:09.472953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e24c17-9048-4cbd-852a-36ab2e828317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:09.473587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e24c17-9048-4cbd-852a-36ab2e828317 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:09.474683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e24c17-9048-4cbd-852a-36ab2e828317 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:09.474955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e24c17-9048-4cbd-852a-36ab2e828317 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:09.475584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e24c17-9048-4cbd-852a-36ab2e828317 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:09.477156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e24c17-9048-4cbd-852a-36ab2e828317 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:09.482126 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:09.486930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1303/2557] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:09 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:09.491782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:09.506805 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1b6823-e678-4cd4-bbbc-b99ecbfae22d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:09.512882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:09.513983 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:09.514739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:09.515075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] PATH_INFO: `/v3/projects/7a8dc35c72f44d6f88020e6d44147da8/users/6452014708c04c6aac55c7b92aae554a/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:09.517111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a8dc35c72f44d6f88020e6d44147da8, user_id=6452014708c04c6aac55c7b92aae554a, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:09.519900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:09.530905 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1255/2558] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:09 2022] PUT /identity/v3/projects/7a8dc35c72f44d6f88020e6d44147da8/users/6452014708c04c6aac55c7b92aae554a/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:09.543038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537fff2-445f-4b77-9091-bbd2277e40ff None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:09.543545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537fff2-445f-4b77-9091-bbd2277e40ff None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:09.543837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7537fff2-445f-4b77-9091-bbd2277e40ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:09.576056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7537fff2-445f-4b77-9091-bbd2277e40ff None None] MFA Rules not processed for user `6452014708c04c6aac55c7b92aae554a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:09.576056 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7537fff2-445f-4b77-9091-bbd2277e40ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:09.641651 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1304/2559] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:09 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:00:09.653074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:09.654858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:09.729264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb255bf-1fd2-4ad3-a41c-c644da659eb8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:09.733209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:09.733823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:09.734095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:09.734366 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:09.791824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:09.793323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:09.796820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:09.805147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1256/2560] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:09 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:11.828903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537fff2-445f-4b77-9091-bbd2277e40ff None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:11.839121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7537fff2-445f-4b77-9091-bbd2277e40ff None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:11.842784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:11.843381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:11.843656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:11.843924 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:11.909204 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:11.910682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:11.920474 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1305/2561] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:13.602189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:13.611010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d6f178-f5e9-41d2-b617-97e5d7a84309 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:13.614588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:13.615325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:13.615601 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:13.615867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:13.624600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:13.625992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:13.635292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1257/2562] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:13 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:14.212466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:14.213972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:14.340227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84c4a20-df25-41bf-b2b7-21b71d67b4f7 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:14.351560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21caa800-e7c3-41d2-9444-4aed2a778fab service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:14.351560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21caa800-e7c3-41d2-9444-4aed2a778fab service placement] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:14.351560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21caa800-e7c3-41d2-9444-4aed2a778fab service placement] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:14.351560 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21caa800-e7c3-41d2-9444-4aed2a778fab service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:14.351945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-21caa800-e7c3-41d2-9444-4aed2a778fab service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:14.359567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:14.366345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21caa800-e7c3-41d2-9444-4aed2a778fab service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:14.368154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21caa800-e7c3-41d2-9444-4aed2a778fab service placement] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:14.386473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1306/2563] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:17.827942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:17.829347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:17.896775 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90dabc8-e459-4a4f-8166-c0d5e0a46f21 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:17.901813 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:17.902478 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:17.902760 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:17.903033 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] PATH_INFO: `/v3/users/ba45f4a0f59a4e688879a0d734e484c4` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:17.904152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba45f4a0f59a4e688879a0d734e484c4)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:17.906050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:18.071814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba45f4a0f59a4e688879a0d734e484c4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:18.078759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba45f4a0f59a4e688879a0d734e484c4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:18.079344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba45f4a0f59a4e688879a0d734e484c4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:18.080057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba45f4a0f59a4e688879a0d734e484c4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:18.149727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba45f4a0f59a4e688879a0d734e484c4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:18.167643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1258/2564] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:17 2022] DELETE /identity/v3/users/ba45f4a0f59a4e688879a0d734e484c4 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:18.172623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:18.173011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:18.173011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:18.194390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:18.196134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:18.254756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1307/2565] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:18.265257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:18.266647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:18.337515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:18.345520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bcf889-199e-49e3-81b8-f268635aa5e5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:18.349226 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:18.349830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:18.350125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:18.350374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:18.359400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:18.360931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:18.370374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1259/2566] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:18.768576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:18.771163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:18.781997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d7677-8a00-4bfa-9edd-9224169962cb None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:18.786674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:18.787310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:18.787592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:18.787865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] PATH_INFO: `/v3/projects/597b9561720a496496d4793b898811cb` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:18.789197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=597b9561720a496496d4793b898811cb)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:18.791176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:18.905231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '597b9561720a496496d4793b898811cb', 'request_id': 'req-2ae69473-5629-4725-8515-b4a6ce725a8a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:18.912273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '597b9561720a496496d4793b898811cb', 'request_id': 'req-2ae69473-5629-4725-8515-b4a6ce725a8a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:18.990137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '597b9561720a496496d4793b898811cb', 'request_id': 'req-2ae69473-5629-4725-8515-b4a6ce725a8a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.001986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] The token cache is being invalidate because project 597b9561720a496496d4793b898811cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:19.008673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.008673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.008673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1308/2567] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:18 2022] DELETE /identity/v3/projects/597b9561720a496496d4793b898811cb => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:19.009488 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:19.019536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a77014-48ef-4dc9-9140-0374c8916383 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:19.078555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:19.079188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:19.079463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:19.079736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] PATH_INFO: `/v3/users/fb64f0d23a53492b977ae0fc91ff13e8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:19.080858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb64f0d23a53492b977ae0fc91ff13e8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:19.082879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:19.306748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb64f0d23a53492b977ae0fc91ff13e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.313753 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb64f0d23a53492b977ae0fc91ff13e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.314454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb64f0d23a53492b977ae0fc91ff13e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.315061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb64f0d23a53492b977ae0fc91ff13e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.379327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f2319071-9293-485b-a1f4-e5bcad11ffe3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb64f0d23a53492b977ae0fc91ff13e8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:19.402396 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1260/2568] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:19 2022] DELETE /identity/v3/users/fb64f0d23a53492b977ae0fc91ff13e8 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:19.813512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:19.815071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:19.828777 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae69473-5629-4725-8515-b4a6ce725a8a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:19.889190 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:19.889823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:19.890098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:19.890368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] PATH_INFO: `/v3/projects/17d76effb63448c0bd12f08b454c1006` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:19.891531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17d76effb63448c0bd12f08b454c1006)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:19.893448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:19.915963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:20.017099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17d76effb63448c0bd12f08b454c1006', 'request_id': 'req-e4926f74-e071-4ba2-8bc0-a296eb5d1293'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:20.024189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17d76effb63448c0bd12f08b454c1006', 'request_id': 'req-e4926f74-e071-4ba2-8bc0-a296eb5d1293'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:20.069810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17d76effb63448c0bd12f08b454c1006', 'request_id': 'req-e4926f74-e071-4ba2-8bc0-a296eb5d1293'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:20.082394 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] The token cache is being invalidate because project 17d76effb63448c0bd12f08b454c1006 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:20.082714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:20.083362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e4926f74-e071-4ba2-8bc0-a296eb5d1293 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:20.084857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1309/2569] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:19 2022] DELETE /identity/v3/projects/17d76effb63448c0bd12f08b454c1006 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:20.090353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ded66-09c8-430c-ba6f-72dd8fca1498 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.090855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ded66-09c8-430c-ba6f-72dd8fca1498 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.090951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ded66-09c8-430c-ba6f-72dd8fca1498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.111181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-cf1ded66-09c8-430c-ba6f-72dd8fca1498 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:20.112393 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cf1ded66-09c8-430c-ba6f-72dd8fca1498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:20.170148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1261/2570] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:20.177935 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d54e54-c1a2-4a2b-a524-ace0b85ec7f0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.178255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d54e54-c1a2-4a2b-a524-ace0b85ec7f0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.178579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d54e54-c1a2-4a2b-a524-ace0b85ec7f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.199680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-27d54e54-c1a2-4a2b-a524-ace0b85ec7f0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:20.200888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-27d54e54-c1a2-4a2b-a524-ace0b85ec7f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:20.208682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1310/2571] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:20.213475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1ded66-09c8-430c-ba6f-72dd8fca1498 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:20.223636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1ded66-09c8-430c-ba6f-72dd8fca1498 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:20.228603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:20.230682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.230682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.230682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.230682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:00:20.231048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:20.232380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:20.233918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:20.244925 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1262/2572] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:00:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:00:20.251198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d54e54-c1a2-4a2b-a524-ace0b85ec7f0 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:20.261149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d54e54-c1a2-4a2b-a524-ace0b85ec7f0 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:20.266137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:20.266801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.267078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.267349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.267972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:20.270071 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:20.369653 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1311/2573] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:20 2022] POST /identity/v3/projects => generated 365 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:20.375520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:20.385994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e45534a-e85a-4f37-b490-c640b978f791 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:20.443445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:20.444054 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.444328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.444830 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.446373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:20.448324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:20.594519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1263/2574] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:20 2022] POST /identity/v3/users => generated 283 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:20.598918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:20.609836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081e3def-2dfb-4ece-93db-c38d21c16580 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:20.614525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:20.615129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.615406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.615675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.616316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:20.618603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:20.628647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1312/2575] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:20.633233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:20.643096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59516d62-03c0-4c7f-9383-62d99bea43f6 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:20.647847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77227616-e435-415b-a9c2-ff7319af2061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:20.648568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77227616-e435-415b-a9c2-ff7319af2061 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.648961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77227616-e435-415b-a9c2-ff7319af2061 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.649245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77227616-e435-415b-a9c2-ff7319af2061 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.649878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77227616-e435-415b-a9c2-ff7319af2061 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:20.651595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77227616-e435-415b-a9c2-ff7319af2061 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:20.661970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1264/2576] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:20.666646 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:20.677665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0af904-36f3-48c9-a73a-e97a112b126e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:20.683953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:20.685081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.685081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.685269 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] PATH_INFO: `/v3/projects/0b4f4a7579a04cdba737240c36533194/users/1638363bc6c4439bae1a1d9f93200e82/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.703463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b4f4a7579a04cdba737240c36533194, user_id=1638363bc6c4439bae1a1d9f93200e82, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:20.708096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:20.794785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1313/2577] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:20 2022] PUT /identity/v3/projects/0b4f4a7579a04cdba737240c36533194/users/1638363bc6c4439bae1a1d9f93200e82/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:20.807147 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5aeef1-b6cc-4240-8203-ac2d61ea9da2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.807427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5aeef1-b6cc-4240-8203-ac2d61ea9da2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.807724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5aeef1-b6cc-4240-8203-ac2d61ea9da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.834628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-3b5aeef1-b6cc-4240-8203-ac2d61ea9da2 None None] MFA Rules not processed for user `1638363bc6c4439bae1a1d9f93200e82`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:20.835861 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3b5aeef1-b6cc-4240-8203-ac2d61ea9da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:20.902280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1265/2578] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:20 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:20.907860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:20.980702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddb56fb-e703-4017-a305-5c2cc076a637 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:20.985699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:20.986678 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:20.986678 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:20.986874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:20.987429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:20.989323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.020318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3d0ff2-2835-4f76-ba0e-290c35d527e5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.021134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3d0ff2-2835-4f76-ba0e-290c35d527e5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.021490 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3d0ff2-2835-4f76-ba0e-290c35d527e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.052875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6f3d0ff2-2835-4f76-ba0e-290c35d527e5 None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:21.054178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6f3d0ff2-2835-4f76-ba0e-290c35d527e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:21.084196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1314/2579] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:20 2022] POST /identity/v3/projects => generated 365 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:21.084938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.086752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:21.108250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:21.120170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:21.123759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1266/2580] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Oct 4 09:00:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:00:21.124579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3d0ff2-2835-4f76-ba0e-290c35d527e5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.144614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3d0ff2-2835-4f76-ba0e-290c35d527e5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.162863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4633dc8-c20a-4998-96c0-19bfdc0b5f76 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.168511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.171180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.171180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.171180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] PATH_INFO: `/v3/users/47c0b4f34d6349cd9fd460996f52b8f8` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.171180 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47c0b4f34d6349cd9fd460996f52b8f8)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.173122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.199453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.200060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.200340 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.200604 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.201224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.203198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.300048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1267/2581] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:21 2022] POST /identity/v3/users => generated 289 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:21.300890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.305430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:21.371115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90da319-896d-4383-8daa-7cec6013ef4c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.376376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.377003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.377277 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.377671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.404583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47c0b4f34d6349cd9fd460996f52b8f8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:21.412609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47c0b4f34d6349cd9fd460996f52b8f8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:21.423656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47c0b4f34d6349cd9fd460996f52b8f8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:21.423656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47c0b4f34d6349cd9fd460996f52b8f8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:21.438132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.439801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.448417 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1268/2582] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 09:00:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 09:00:21.449200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.459028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7089594c-d2b5-4609-873d-247432c0b15e service swift] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.463991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.464597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.464820 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.465146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.465768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.467376 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.490364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1269/2583] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:21 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:21.497589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7fdc8524-00fc-4f05-9b5e-766dc8eb0236 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47c0b4f34d6349cd9fd460996f52b8f8'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:21.500334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.502655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:21.518503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bd32a-9270-4364-a987-0d8a436e619d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.523706 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.524573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.524759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.525053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] PATH_INFO: `/v3/projects/75e4a2374af74c3a98dd86ab5cecc086/users/05734f595f784424a8c8fb1ea0099d97/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.530384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1315/2584] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:21 2022] DELETE /identity/v3/users/47c0b4f34d6349cd9fd460996f52b8f8 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:21.536582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.536978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.537745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.539815 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:21.551468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=75e4a2374af74c3a98dd86ab5cecc086, user_id=05734f595f784424a8c8fb1ea0099d97, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.554198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.565152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:21.566864 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:21.628631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1270/2585] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:21 2022] PUT /identity/v3/projects/75e4a2374af74c3a98dd86ab5cecc086/users/05734f595f784424a8c8fb1ea0099d97/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:21.632567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.644130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1316/2586] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:21.654790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.656179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:21.719169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:21.723304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c737bff1-f28b-4939-ad15-8af0746154d1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.725823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6637d-f735-4063-9fb0-f275c99671fb admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.732464 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.733499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.734052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.734684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.735995 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.738270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.767491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1271/2587] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:21 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:21.768528 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:21.787028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.787825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.788214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.789824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.793073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e43110-3f9b-45e3-b57c-b190416940d5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:21.802388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36ba600-1c3e-46bd-b95a-80b8c9f52788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:21.802388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ba600-1c3e-46bd-b95a-80b8c9f52788 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.802388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ba600-1c3e-46bd-b95a-80b8c9f52788 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.802388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ba600-1c3e-46bd-b95a-80b8c9f52788 admin admin] PATH_INFO: `/v3/projects/75e4a2374af74c3a98dd86ab5cecc086/users/05734f595f784424a8c8fb1ea0099d97/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.809899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ba600-1c3e-46bd-b95a-80b8c9f52788 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75e4a2374af74c3a98dd86ab5cecc086, user_id=05734f595f784424a8c8fb1ea0099d97, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.815475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:21.816643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ba600-1c3e-46bd-b95a-80b8c9f52788 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.817675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:21.842693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1272/2588] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:21 2022] PUT /identity/v3/projects/75e4a2374af74c3a98dd86ab5cecc086/users/05734f595f784424a8c8fb1ea0099d97/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:21.848527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1317/2589] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:00:21.853656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:21.854006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:21.854343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:21.908616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] MFA Rules not processed for user `05734f595f784424a8c8fb1ea0099d97`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:21.909940 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:22.014445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1273/2590] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:21 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:22.027514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:22.029149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:22.236217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52562572-5934-45ab-9a4b-88f1d26ded24 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:22.369218 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:22.369981 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:22.370326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:22.370799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:22.385961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:22.392428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:22.403197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1318/2591] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:22 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:00:22.433044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:22.435231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:22.542180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:22.549373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372a3a6-58e1-438a-bf18-1c87664ece04 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:22.557814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:22.558956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:22.558956 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:22.559352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] PATH_INFO: `/v3/projects/422aef83dc3a4d0f91707a491f9993f6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:22.560541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] RBAC: Authorizing `identity:delete_project(project_id=422aef83dc3a4d0f91707a491f9993f6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:22.562503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:22.671613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:22.682315 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db1a6e9-4d57-4e20-a4f8-caea0d9e34c3 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:22.686187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:22.686825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:22.687103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:22.687369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:22.689117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:22.737024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '422aef83dc3a4d0f91707a491f9993f6', 'request_id': 'req-90651d3a-75d2-4bf5-b868-50279672ef5d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:22.749606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:22.765562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:22.767167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:22.783892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1319/2592] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:22 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:00:22.843016 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '422aef83dc3a4d0f91707a491f9993f6', 'request_id': 'req-90651d3a-75d2-4bf5-b868-50279672ef5d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:22.850385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '422aef83dc3a4d0f91707a491f9993f6', 'request_id': 'req-90651d3a-75d2-4bf5-b868-50279672ef5d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:22.863875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] The token cache is being invalidate because project 422aef83dc3a4d0f91707a491f9993f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:22.864168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:22.864947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:22.868470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1274/2593] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:22 2022] DELETE /identity/v3/projects/422aef83dc3a4d0f91707a491f9993f6 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:22.872803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:22.885506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3be70-f43a-4734-ba5f-8451316da37b service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:22.970215 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:22.972373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:22.973491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:22.974379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] PATH_INFO: `/v3/users/dd908eebceb14ab8801569458b8a9311` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:22.978522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd908eebceb14ab8801569458b8a9311)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:22.980547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:23.212485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd908eebceb14ab8801569458b8a9311'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:23.220990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd908eebceb14ab8801569458b8a9311'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:23.221715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd908eebceb14ab8801569458b8a9311'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:23.222338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd908eebceb14ab8801569458b8a9311'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:23.327761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd908eebceb14ab8801569458b8a9311'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:23.365817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1320/2594] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:22 2022] DELETE /identity/v3/users/dd908eebceb14ab8801569458b8a9311 => generated 0 bytes in 491 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:23.896095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:23.897683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:23.930014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90651d3a-75d2-4bf5-b868-50279672ef5d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:24.002354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:24.003293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:24.003583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:24.003847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] PATH_INFO: `/v3/projects/051583a767f84166ada94c1ea9d5bdac` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:24.005036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=051583a767f84166ada94c1ea9d5bdac)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:24.007218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:24.150847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '051583a767f84166ada94c1ea9d5bdac', 'request_id': 'req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.246115 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '051583a767f84166ada94c1ea9d5bdac', 'request_id': 'req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.255608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '051583a767f84166ada94c1ea9d5bdac', 'request_id': 'req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.271611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] The token cache is being invalidate because project 051583a767f84166ada94c1ea9d5bdac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:24.272024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.272569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.278761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1275/2595] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:23 2022] DELETE /identity/v3/projects/051583a767f84166ada94c1ea9d5bdac => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:24.279148 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:24.281074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:24.296838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ce8e7b-6ac3-421b-a511-908a663b9fec admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:24.389418 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:24.390064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:24.390064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:24.390349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] PATH_INFO: `/v3/users/6452014708c04c6aac55c7b92aae554a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:24.391608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=6452014708c04c6aac55c7b92aae554a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:24.393582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:24.411103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:24.597796 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6452014708c04c6aac55c7b92aae554a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.606886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6452014708c04c6aac55c7b92aae554a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.607827 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6452014708c04c6aac55c7b92aae554a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.608435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6452014708c04c6aac55c7b92aae554a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.886087 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dee29deb-a011-41e0-9ca9-b7df4e5a9bfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6452014708c04c6aac55c7b92aae554a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:24.904779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1321/2596] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:24 2022] DELETE /identity/v3/users/6452014708c04c6aac55c7b92aae554a => generated 0 bytes in 626 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:25.400260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:25.402456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:25.423855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c952ed4c-e30d-4b43-a1c8-00fda0111b3c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:25.450585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:25.543302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:25.544034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:25.544380 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:25.544704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] PATH_INFO: `/v3/projects/7a8dc35c72f44d6f88020e6d44147da8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:25.545888 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a8dc35c72f44d6f88020e6d44147da8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:25.548005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:25.734271 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a8dc35c72f44d6f88020e6d44147da8', 'request_id': 'req-527106d0-b923-4718-945a-0bb9fce5465c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:25.826683 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a8dc35c72f44d6f88020e6d44147da8', 'request_id': 'req-527106d0-b923-4718-945a-0bb9fce5465c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:25.835087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a8dc35c72f44d6f88020e6d44147da8', 'request_id': 'req-527106d0-b923-4718-945a-0bb9fce5465c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:25.847630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] The token cache is being invalidate because project 7a8dc35c72f44d6f88020e6d44147da8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:25.847919 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:25.848699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-527106d0-b923-4718-945a-0bb9fce5465c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:25.850191 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1276/2597] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:25 2022] DELETE /identity/v3/projects/7a8dc35c72f44d6f88020e6d44147da8 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:25.858617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069d4ce-6cfe-45ac-bd97-4f26588efa39 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:25.862655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069d4ce-6cfe-45ac-bd97-4f26588efa39 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:25.862655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069d4ce-6cfe-45ac-bd97-4f26588efa39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:25.881367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a069d4ce-6cfe-45ac-bd97-4f26588efa39 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:25.883337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a069d4ce-6cfe-45ac-bd97-4f26588efa39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:25.963972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1322/2598] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:25.972274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945e936-fe75-4ac0-9196-815bd67fd8a0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:25.972654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945e936-fe75-4ac0-9196-815bd67fd8a0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:25.972962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945e936-fe75-4ac0-9196-815bd67fd8a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.000132 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5945e936-fe75-4ac0-9196-815bd67fd8a0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:26.001371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5945e936-fe75-4ac0-9196-815bd67fd8a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:26.013024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1277/2599] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:26.017521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a069d4ce-6cfe-45ac-bd97-4f26588efa39 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:26.029421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a069d4ce-6cfe-45ac-bd97-4f26588efa39 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:26.036547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:26.037220 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.037492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.037756 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.038438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:00:26.038750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:26.041090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:26.056370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1323/2600] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:00:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:00:26.066798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945e936-fe75-4ac0-9196-815bd67fd8a0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:26.077640 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945e936-fe75-4ac0-9196-815bd67fd8a0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:26.082934 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:26.083549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.083828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.084125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.084772 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:26.087652 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:26.178777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1278/2601] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:26 2022] POST /identity/v3/projects => generated 371 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:26.182675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:26.196715 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e15700-e52c-4322-b970-11f23aad4423 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:26.282928 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:26.283635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.283987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.284348 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.285048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:26.287157 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:26.437673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:26.444149 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1324/2602] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:26 2022] POST /identity/v3/users => generated 286 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:26.448599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:26.460624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc7ba71-9465-45c4-9679-a250208533a6 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:26.465972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:26.466778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.467067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.467342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.469289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:26.470932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:26.482993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1279/2603] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:26 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:26.495217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:26.508398 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2424744-da25-4b5a-af2b-c3f5c75c732e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:26.513634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd45fa-e480-4a4a-b3ca-a52d21594c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:26.514625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd45fa-e480-4a4a-b3ca-a52d21594c83 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.514909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd45fa-e480-4a4a-b3ca-a52d21594c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.515167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd45fa-e480-4a4a-b3ca-a52d21594c83 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.515800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd45fa-e480-4a4a-b3ca-a52d21594c83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:26.517828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dd45fa-e480-4a4a-b3ca-a52d21594c83 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:26.531072 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1325/2604] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:26 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:26.535715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:26.547180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cf05b2-4b48-49db-b6c9-48fa3c423c4c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:26.556202 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:26.557024 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.557310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.557583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] PATH_INFO: `/v3/projects/bd948607d91d4689b7b169a075aac661/users/c682afa0ab1f401f850514e5baf7633a/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.574225 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd948607d91d4689b7b169a075aac661, user_id=c682afa0ab1f401f850514e5baf7633a, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:26.577031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:26.669739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1280/2605] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:26 2022] PUT /identity/v3/projects/bd948607d91d4689b7b169a075aac661/users/c682afa0ab1f401f850514e5baf7633a/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:26.681400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.681684 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.681973 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.735273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] MFA Rules not processed for user `c682afa0ab1f401f850514e5baf7633a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:26.737360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:26.844927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1326/2606] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:00:26.859431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:26.861382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:26.917364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:26.934635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a8a3c-c0df-465c-9540-c0c1df718e56 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:26.940415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caede7ed-33fd-4893-b3db-e8e1c0f942dd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '146238397f514eada174ab3374be5418', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:26.941502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caede7ed-33fd-4893-b3db-e8e1c0f942dd service glance] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:26.941781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caede7ed-33fd-4893-b3db-e8e1c0f942dd service glance] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:26.942052 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caede7ed-33fd-4893-b3db-e8e1c0f942dd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:26.957403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caede7ed-33fd-4893-b3db-e8e1c0f942dd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:26.958696 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caede7ed-33fd-4893-b3db-e8e1c0f942dd service glance] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:26.968806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1281/2607] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:26 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:34.934360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:34.936042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:35.019855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcfd302-ff16-47c1-b216-536363fe4ef9 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:35.025236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:35.025889 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:35.026378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:35.026705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] PATH_INFO: `/v3/users/1638363bc6c4439bae1a1d9f93200e82` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:35.028613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1638363bc6c4439bae1a1d9f93200e82)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:35.031033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:35.238968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1638363bc6c4439bae1a1d9f93200e82'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:35.254750 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1638363bc6c4439bae1a1d9f93200e82'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:35.255923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1638363bc6c4439bae1a1d9f93200e82'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:35.256683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1638363bc6c4439bae1a1d9f93200e82'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:35.619403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1638363bc6c4439bae1a1d9f93200e82'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:35.632886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:35.652197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1327/2608] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:34 2022] DELETE /identity/v3/users/1638363bc6c4439bae1a1d9f93200e82 => generated 0 bytes in 718 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:35.656489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:35.656991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:35.657162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:35.677123 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:35.678347 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:35.760804 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1282/2609] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:00:35.775044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:35.777674 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:35.841525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f7e607-e0b6-4c81-aa23-4c15ac76cb85 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:35.845383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:35.846106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:35.846518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:35.846907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:35.858030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:35.861659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:35.872441 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1328/2610] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:36.381388 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:36.384057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:36.411103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c4172-8403-40d2-b3ae-b70fce22f28a None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:36.427503 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:36.428803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:36.429118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:36.430178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] PATH_INFO: `/v3/projects/0b4f4a7579a04cdba737240c36533194` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:36.433010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b4f4a7579a04cdba737240c36533194)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:36.437798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:36.576943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b4f4a7579a04cdba737240c36533194', 'request_id': 'req-fce0cfec-2219-4167-a4c0-22a1f970a907'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.682322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b4f4a7579a04cdba737240c36533194', 'request_id': 'req-fce0cfec-2219-4167-a4c0-22a1f970a907'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.691296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b4f4a7579a04cdba737240c36533194', 'request_id': 'req-fce0cfec-2219-4167-a4c0-22a1f970a907'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.717602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] The token cache is being invalidate because project 0b4f4a7579a04cdba737240c36533194 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:36.717982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.718680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.720434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1283/2611] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:36 2022] DELETE /identity/v3/projects/0b4f4a7579a04cdba737240c36533194 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:36.724311 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:36.734457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e9211e-5464-44a6-96b0-9a4f7fa15c69 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:36.801735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:36.803169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:36.803169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:36.803169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] PATH_INFO: `/v3/users/05734f595f784424a8c8fb1ea0099d97` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:36.804124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] RBAC: Authorizing `identity:delete_user(user_id=05734f595f784424a8c8fb1ea0099d97)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:36.807073 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:36.956006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05734f595f784424a8c8fb1ea0099d97'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.965661 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05734f595f784424a8c8fb1ea0099d97'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.966652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05734f595f784424a8c8fb1ea0099d97'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:36.967007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05734f595f784424a8c8fb1ea0099d97'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:37.016791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8ce573da-40ae-47fe-a6cd-2ea58010293b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05734f595f784424a8c8fb1ea0099d97'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:37.045165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1329/2612] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:36 2022] DELETE /identity/v3/users/05734f595f784424a8c8fb1ea0099d97 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:37.498666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:37.506984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:37.552432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0cfec-2219-4167-a4c0-22a1f970a907 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:37.646812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:37.647795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:37.648839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:37.650359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] PATH_INFO: `/v3/projects/75e4a2374af74c3a98dd86ab5cecc086` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:37.652850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] RBAC: Authorizing `identity:delete_project(project_id=75e4a2374af74c3a98dd86ab5cecc086)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:37.655979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:37.681621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:37.808840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75e4a2374af74c3a98dd86ab5cecc086', 'request_id': 'req-37b0a27d-8728-453f-9f32-c1ecb510834a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:37.887007 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75e4a2374af74c3a98dd86ab5cecc086', 'request_id': 'req-37b0a27d-8728-453f-9f32-c1ecb510834a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:37.894647 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75e4a2374af74c3a98dd86ab5cecc086', 'request_id': 'req-37b0a27d-8728-453f-9f32-c1ecb510834a'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:37.906133 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] The token cache is being invalidate because project 75e4a2374af74c3a98dd86ab5cecc086 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:37.906332 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:37.907032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-37b0a27d-8728-453f-9f32-c1ecb510834a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:37.912129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1284/2613] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:37 2022] DELETE /identity/v3/projects/75e4a2374af74c3a98dd86ab5cecc086 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:37.914146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1358ab77-fea8-4b94-9b3f-32bcf769db00 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:37.914657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1358ab77-fea8-4b94-9b3f-32bcf769db00 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:37.914950 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1358ab77-fea8-4b94-9b3f-32bcf769db00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:37.936395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1358ab77-fea8-4b94-9b3f-32bcf769db00 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:37.938261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1358ab77-fea8-4b94-9b3f-32bcf769db00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:37.955592 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:38.008555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1330/2614] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:38.016902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3191a17-8856-47f0-82c0-dd41bbf289a0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.017441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3191a17-8856-47f0-82c0-dd41bbf289a0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.017762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3191a17-8856-47f0-82c0-dd41bbf289a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.041300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-f3191a17-8856-47f0-82c0-dd41bbf289a0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:38.042553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f3191a17-8856-47f0-82c0-dd41bbf289a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:38.052538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1285/2615] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:38.057068 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1358ab77-fea8-4b94-9b3f-32bcf769db00 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:38.068284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1358ab77-fea8-4b94-9b3f-32bcf769db00 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:38.074075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:38.074716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.074996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.075295 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.076063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:00:38.076364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:38.078232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:38.095508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1331/2616] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:00:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:00:38.105253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3191a17-8856-47f0-82c0-dd41bbf289a0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:38.114833 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3191a17-8856-47f0-82c0-dd41bbf289a0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:38.121472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:38.122011 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.122288 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.122587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.123213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:38.125090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:38.236090 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1286/2617] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:38 2022] POST /identity/v3/projects => generated 387 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:38.237412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:38.248705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ea96c4-0021-4a82-918b-54c7a252f446 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:38.320969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:38.321454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.321712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.321988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.322666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:38.324595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:38.453718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1332/2618] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:38 2022] POST /identity/v3/users => generated 294 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:38.458274 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:38.470374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862c378a-1b95-479d-81b1-0b965341fa55 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:38.476980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:38.477911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.478198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.478533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.479771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:38.482083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:38.498812 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1287/2619] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:38 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:38.503007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:38.515956 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39383d0-abd2-440d-acf3-9a81906d4011 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:38.521144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9cab85-0f68-4845-9de4-6e7ba0f65fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:38.521762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cab85-0f68-4845-9de4-6e7ba0f65fef admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.522040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cab85-0f68-4845-9de4-6e7ba0f65fef admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.522314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cab85-0f68-4845-9de4-6e7ba0f65fef admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.523192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9cab85-0f68-4845-9de4-6e7ba0f65fef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:38.525970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9cab85-0f68-4845-9de4-6e7ba0f65fef admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:38.541570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1333/2620] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:38 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:38.545768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:38.558042 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c130c4-6de7-477a-a220-ddce439519d8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:38.562742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:38.563418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.563677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.563949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] PATH_INFO: `/v3/projects/5ca8dc5048144432998c07e1298ae6bf/users/5e56919a994c4c499e15b0e58bedbae6/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.582679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ca8dc5048144432998c07e1298ae6bf, user_id=5e56919a994c4c499e15b0e58bedbae6, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:38.586571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:38.693932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1288/2621] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:38 2022] PUT /identity/v3/projects/5ca8dc5048144432998c07e1298ae6bf/users/5e56919a994c4c499e15b0e58bedbae6/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:38.704741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab7932-8215-4d1c-8b51-bc828bbed74d None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.705054 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab7932-8215-4d1c-8b51-bc828bbed74d None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.705325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab7932-8215-4d1c-8b51-bc828bbed74d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.750748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-45ab7932-8215-4d1c-8b51-bc828bbed74d None None] MFA Rules not processed for user `5e56919a994c4c499e15b0e58bedbae6`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:38.751908 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-45ab7932-8215-4d1c-8b51-bc828bbed74d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:38.886119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1334/2622] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:38 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:00:38.887913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:38.985912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb0e145-03f7-46a3-8ac8-95bc5acfbc5f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:38.991365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:38.992619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:38.992965 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:38.993304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:38.994050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:38.996236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:39.138043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1289/2623] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:38 2022] POST /identity/v3/projects => generated 385 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:39.143207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ab7932-8215-4d1c-8b51-bc828bbed74d None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:39.163454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ab7932-8215-4d1c-8b51-bc828bbed74d None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:39.183921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:39.258025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:39.258874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:39.259298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:39.259819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:39.260570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:39.262879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:39.363238 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1335/2624] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:39 2022] POST /identity/v3/users => generated 293 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:39.369680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:39.397592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c874ed48-986b-47b7-8153-644c768a8ada admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:39.406468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:39.406468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:39.406468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:39.406468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:39.412535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:39.412535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:39.420139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1290/2625] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:39 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:39.426468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:39.439613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae3a1ba-bc3e-4b94-b298-16dd8f6e2de7 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:39.445024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02547f0-0638-4f5a-8039-2db137244305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:39.445815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02547f0-0638-4f5a-8039-2db137244305 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:39.446011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02547f0-0638-4f5a-8039-2db137244305 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:39.446288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02547f0-0638-4f5a-8039-2db137244305 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:39.447151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02547f0-0638-4f5a-8039-2db137244305 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:39.448822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02547f0-0638-4f5a-8039-2db137244305 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:39.464192 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1336/2626] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:39 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:39.473822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:39.486890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c647c-8841-4cb9-95aa-42daf3afff25 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:39.496217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:39.496933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:39.497243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:39.497493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] PATH_INFO: `/v3/projects/619278d625a947efabbd53f60ce6628f/users/8a28a3ab2b7a413d88e86a1f92a40b93/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:39.504019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:39.528489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=619278d625a947efabbd53f60ce6628f, user_id=8a28a3ab2b7a413d88e86a1f92a40b93, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:39.532116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:40.031300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1291/2627] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:39 2022] PUT /identity/v3/projects/619278d625a947efabbd53f60ce6628f/users/8a28a3ab2b7a413d88e86a1f92a40b93/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 563 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:40.042475 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41767e-4100-4e44-9f8a-9ee566ce7d16 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:40.042785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41767e-4100-4e44-9f8a-9ee566ce7d16 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:40.043058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41767e-4100-4e44-9f8a-9ee566ce7d16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:40.072132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-3c41767e-4100-4e44-9f8a-9ee566ce7d16 None None] MFA Rules not processed for user `8a28a3ab2b7a413d88e86a1f92a40b93`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:40.073429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3c41767e-4100-4e44-9f8a-9ee566ce7d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:40.167748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1337/2628] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:40 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:40.172564 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:40.266028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd4dc7f-1e5f-4b9f-9eae-d389f694de6b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:40.279918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:40.280649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:40.280935 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:40.281210 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:40.292321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:40.294373 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:40.406050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1292/2629] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:40 2022] POST /identity/v3/projects => generated 387 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:40.410108 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41767e-4100-4e44-9f8a-9ee566ce7d16 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:40.423506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41767e-4100-4e44-9f8a-9ee566ce7d16 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:40.541103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:40.542116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:40.542783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:40.543136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:40.544203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:40.547569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:40.649447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1338/2630] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:40 2022] POST /identity/v3/users => generated 300 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:40.655845 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:40.668953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62636c62-5fdf-45c9-9d1e-770951f767b8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:40.674085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:40.674869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:40.675149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:40.675420 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:40.676074 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:40.677868 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:40.693418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1293/2631] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:40.695826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:40.706890 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c1112-57b3-411d-acb2-2eb4b65224a0 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:40.713093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:40.713873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:40.714256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:40.714858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] PATH_INFO: `/v3/projects/8f5e959e140c421883424057d7631d89/users/efd194163aa9419e9014ef200d613fbc/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:40.729987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:40.734479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f5e959e140c421883424057d7631d89, user_id=efd194163aa9419e9014ef200d613fbc, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:40.737332 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:40.830957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1339/2632] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:40 2022] PUT /identity/v3/projects/8f5e959e140c421883424057d7631d89/users/efd194163aa9419e9014ef200d613fbc/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:40.834855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:40.853920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81717fb-1304-4e0a-a030-f0ed9d1820a7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:40.931701 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d9c3d7-d0c4-4ba7-a8a5-1d03d04d680d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:40.932441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d9c3d7-d0c4-4ba7-a8a5-1d03d04d680d admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:40.932715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d9c3d7-d0c4-4ba7-a8a5-1d03d04d680d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:40.933139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d9c3d7-d0c4-4ba7-a8a5-1d03d04d680d admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:40.933789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d9c3d7-d0c4-4ba7-a8a5-1d03d04d680d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:40.935809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d9c3d7-d0c4-4ba7-a8a5-1d03d04d680d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:40.952134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1294/2633] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:00:40.953213 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:40.988629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a83ab-8f80-4f07-bbcc-5a0ee6d026f4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:40.996574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:40.997281 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:40.997709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:40.997988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] PATH_INFO: `/v3/projects/8f5e959e140c421883424057d7631d89/users/efd194163aa9419e9014ef200d613fbc/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:40.999975 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f5e959e140c421883424057d7631d89, user_id=efd194163aa9419e9014ef200d613fbc, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:41.002839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:41.016378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1340/2634] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:40 2022] PUT /identity/v3/projects/8f5e959e140c421883424057d7631d89/users/efd194163aa9419e9014ef200d613fbc/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:41.030383 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:41.030798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:41.031180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:41.069353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] MFA Rules not processed for user `efd194163aa9419e9014ef200d613fbc`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:41.071573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:41.145799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:41.175491 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1295/2635] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:41 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:41.191600 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:41.193537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:41.262870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851867dd-b276-4362-888d-78f4ef0fbbab admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:41.268651 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcde3db4-4e02-446a-9b39-6e983e826a82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:41.269291 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcde3db4-4e02-446a-9b39-6e983e826a82 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:41.269570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcde3db4-4e02-446a-9b39-6e983e826a82 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:41.269844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcde3db4-4e02-446a-9b39-6e983e826a82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:41.281014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcde3db4-4e02-446a-9b39-6e983e826a82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:41.282480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcde3db4-4e02-446a-9b39-6e983e826a82 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:41.293407 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1341/2636] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:41 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:42.449742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:42.452263 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:42.478351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89de81d3-2460-47b7-a147-6b7dad400470 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:42.484677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:42.485962 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:42.486251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:42.486918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:42.564992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:42.566338 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:42.576618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1296/2637] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:42 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:47.284044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36972753-106f-4b4c-9c38-c622f112553a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:47.284633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36972753-106f-4b4c-9c38-c622f112553a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:47.284633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36972753-106f-4b4c-9c38-c622f112553a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:47.306593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-36972753-106f-4b4c-9c38-c622f112553a None None] MFA Rules not processed for user `c9cff7d749314dbc94645a7884792496`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:47.307877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-36972753-106f-4b4c-9c38-c622f112553a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:47.389765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1342/2638] 10.222.0.27 () {58 vars in 967 bytes} [Tue Oct 4 09:00:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:47.403429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:47.405807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:47.485261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df119d-9984-4ae2-9d97-33c4d0953ccb service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:47.490320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '085a0cc3f5cb477989e6e75adea7c53a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:47.490998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:47.491271 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:47.491557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:47.500214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:47.501806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:47.509847 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1297/2639] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Oct 4 09:00:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 09:00:50.505268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36972753-106f-4b4c-9c38-c622f112553a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:50.506914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-36972753-106f-4b4c-9c38-c622f112553a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:50.593819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36972753-106f-4b4c-9c38-c622f112553a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:50.598602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:50.599324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:50.599744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:50.600018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] PATH_INFO: `/v3/users/6027a9e1cfe74099a92eb3b39ce6361b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:50.601194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6027a9e1cfe74099a92eb3b39ce6361b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:50.603305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:50.705711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:50.811525 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6027a9e1cfe74099a92eb3b39ce6361b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:50.818388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6027a9e1cfe74099a92eb3b39ce6361b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:50.819082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6027a9e1cfe74099a92eb3b39ce6361b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:50.819686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6027a9e1cfe74099a92eb3b39ce6361b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:50.910206 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6027a9e1cfe74099a92eb3b39ce6361b'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:50.930182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1343/2640] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:50 2022] DELETE /identity/v3/users/6027a9e1cfe74099a92eb3b39ce6361b => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:50.997121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:51.000402 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9416ec25-9940-4c9d-b7b9-e6a97679afdd service swift] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:51.053584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388a2ef7-af76-496a-bba0-04bf3af5ce94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:51.054200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388a2ef7-af76-496a-bba0-04bf3af5ce94 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.054512 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388a2ef7-af76-496a-bba0-04bf3af5ce94 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.054789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388a2ef7-af76-496a-bba0-04bf3af5ce94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.056547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-388a2ef7-af76-496a-bba0-04bf3af5ce94 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:51.080975 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=100 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:51.124185 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388a2ef7-af76-496a-bba0-04bf3af5ce94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:51.125662 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388a2ef7-af76-496a-bba0-04bf3af5ce94 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:51.134949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1298/2641] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:50 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:51.213463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:51.215051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:51.286458 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbc23d4-7e73-474e-8d49-539eefa9ebc6 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:51.291116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:51.291753 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.292159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.292431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] PATH_INFO: `/v3/projects/f9e5915b40b84a859f5ab0fb5892c973` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.293545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9e5915b40b84a859f5ab0fb5892c973)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:51.295830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:51.425623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9e5915b40b84a859f5ab0fb5892c973', 'request_id': 'req-9472022e-b2d9-431c-9289-3561026518e7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:51.431947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9e5915b40b84a859f5ab0fb5892c973', 'request_id': 'req-9472022e-b2d9-431c-9289-3561026518e7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:51.487221 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9e5915b40b84a859f5ab0fb5892c973', 'request_id': 'req-9472022e-b2d9-431c-9289-3561026518e7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:51.498711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] The token cache is being invalidate because project f9e5915b40b84a859f5ab0fb5892c973 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:51.499718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:51.499718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-9472022e-b2d9-431c-9289-3561026518e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:51.501212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1344/2642] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:51 2022] DELETE /identity/v3/projects/f9e5915b40b84a859f5ab0fb5892c973 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:51.506249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920325f-92a9-4ed8-8e4b-81d4f5e931b7 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.506561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920325f-92a9-4ed8-8e4b-81d4f5e931b7 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.506828 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920325f-92a9-4ed8-8e4b-81d4f5e931b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.526802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5920325f-92a9-4ed8-8e4b-81d4f5e931b7 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:51.527989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5920325f-92a9-4ed8-8e4b-81d4f5e931b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:51.592933 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1299/2643] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:51.600618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3e24c-0c02-4916-bf4a-1d893f41b72a None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.601063 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3e24c-0c02-4916-bf4a-1d893f41b72a None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.601224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3e24c-0c02-4916-bf4a-1d893f41b72a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.623610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-53a3e24c-0c02-4916-bf4a-1d893f41b72a None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:51.627747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-53a3e24c-0c02-4916-bf4a-1d893f41b72a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:51.633786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1345/2644] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:51.638001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5920325f-92a9-4ed8-8e4b-81d4f5e931b7 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:51.648761 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5920325f-92a9-4ed8-8e4b-81d4f5e931b7 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:51.656386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:51.656386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.656386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.656386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.656862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:00:51.656862 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:51.657267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:51.669432 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1300/2645] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:00:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 09:00:51.673702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a3e24c-0c02-4916-bf4a-1d893f41b72a None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:51.683178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a3e24c-0c02-4916-bf4a-1d893f41b72a None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:51.688690 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:51.689277 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.689557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.689830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.690497 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:51.692487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:51.797351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1346/2646] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:51 2022] POST /identity/v3/projects => generated 377 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:51.801433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:51.812249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6142a98c-3c33-44b2-9d13-5ea07617c6d9 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:51.870044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:51.871303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.871303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.871303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.871801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:51.874524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:51.951474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1301/2647] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:51 2022] POST /identity/v3/users => generated 289 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:00:51.955717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:51.962605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:51.966641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f39b66-8666-4698-bcfd-3c648fdef155 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:51.971327 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:51.971918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:51.972194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:51.972462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:51.973123 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:51.974783 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:51.985283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1347/2648] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:51.989218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:52.000008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0806a6f-0301-4879-b1f3-cdb816718553 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:52.004734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e90ee24-3346-4710-a3e0-f5c0a6f70767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:52.005405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e90ee24-3346-4710-a3e0-f5c0a6f70767 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:52.005685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e90ee24-3346-4710-a3e0-f5c0a6f70767 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:52.005953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e90ee24-3346-4710-a3e0-f5c0a6f70767 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:52.006628 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e90ee24-3346-4710-a3e0-f5c0a6f70767 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:52.008201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e90ee24-3346-4710-a3e0-f5c0a6f70767 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:52.020642 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1302/2649] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:52.024096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:52.044237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66771774-104b-48d3-9453-ff7592c29925 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:52.049597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:52.050297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:52.050609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:52.050915 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] PATH_INFO: `/v3/projects/07b9220d86234adc83ef7fbe0d4d834d/users/471299b556bd4c22b51defc14b49d8eb/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:52.068117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] RBAC: Authorizing `identity:create_grant(project_id=07b9220d86234adc83ef7fbe0d4d834d, user_id=471299b556bd4c22b51defc14b49d8eb, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:52.070882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:52.139132 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1348/2650] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:52 2022] PUT /identity/v3/projects/07b9220d86234adc83ef7fbe0d4d834d/users/471299b556bd4c22b51defc14b49d8eb/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:52.144293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:52.144575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:52.144851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:52.164963 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:52.166178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:52.191467 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:52.233032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1303/2651] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:52.244718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:52.246454 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:52.307595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d5827-092c-4297-a19d-515d2a6e284c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:52.311284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98654294-3062-40fd-b1b2-cce0ae748c1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:52.311907 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98654294-3062-40fd-b1b2-cce0ae748c1e service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:52.312165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98654294-3062-40fd-b1b2-cce0ae748c1e service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:52.312509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98654294-3062-40fd-b1b2-cce0ae748c1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:52.321514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98654294-3062-40fd-b1b2-cce0ae748c1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:52.322988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98654294-3062-40fd-b1b2-cce0ae748c1e service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:52.332825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1349/2652] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:55.713428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:55.715228 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:55.729081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50a3c5e-8a3e-4db4-90ad-6c565ed3fe83 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:55.736149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:55.738630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:55.738630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:55.738630 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] PATH_INFO: `/v3/users/5e56919a994c4c499e15b0e58bedbae6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:55.739658 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e56919a994c4c499e15b0e58bedbae6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:55.742046 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:55.958908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e56919a994c4c499e15b0e58bedbae6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:55.967942 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e56919a994c4c499e15b0e58bedbae6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:55.968737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e56919a994c4c499e15b0e58bedbae6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:55.969676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e56919a994c4c499e15b0e58bedbae6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:56.046602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e56919a994c4c499e15b0e58bedbae6'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:56.077961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1304/2653] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:55 2022] DELETE /identity/v3/users/5e56919a994c4c499e15b0e58bedbae6 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:56.082622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:56.083040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:56.083169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:56.103363 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:56.104561 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:56.175512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1350/2654] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:56.186866 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:56.188539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:56.268025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a58fd6-2dff-44f8-b01e-9f8c377d1619 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:56.272127 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:56.272703 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:56.272973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:56.273303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:56.286493 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:56.290885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:56.312283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1305/2655] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:00:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:00:57.038109 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:57.040412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:57.071383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94dccf1-e2c0-4a20-97f9-34a8573a47d5 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:57.080017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:57.081024 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:57.081603 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:57.082059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] PATH_INFO: `/v3/projects/5ca8dc5048144432998c07e1298ae6bf` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:57.086202 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ca8dc5048144432998c07e1298ae6bf)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:57.087677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:57.248460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ca8dc5048144432998c07e1298ae6bf', 'request_id': 'req-7ec2a45c-6401-495e-b3fb-b0703813ff93'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.255955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ca8dc5048144432998c07e1298ae6bf', 'request_id': 'req-7ec2a45c-6401-495e-b3fb-b0703813ff93'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.322367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ca8dc5048144432998c07e1298ae6bf', 'request_id': 'req-7ec2a45c-6401-495e-b3fb-b0703813ff93'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.332194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:57.339187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] The token cache is being invalidate because project 5ca8dc5048144432998c07e1298ae6bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:57.339462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.340201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.343002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1351/2656] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:57 2022] DELETE /identity/v3/projects/5ca8dc5048144432998c07e1298ae6bf => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:57.347443 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:57.382852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4eeccd-4655-45cd-82c5-4270b6aeb4e4 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:57.453293 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:57.480612 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:57.481271 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:57.481661 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:57.482044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] PATH_INFO: `/v3/users/8a28a3ab2b7a413d88e86a1f92a40b93` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:57.483494 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a28a3ab2b7a413d88e86a1f92a40b93)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:57.485767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:57.707584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a28a3ab2b7a413d88e86a1f92a40b93'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.714748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a28a3ab2b7a413d88e86a1f92a40b93'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.715719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a28a3ab2b7a413d88e86a1f92a40b93'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.716968 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a28a3ab2b7a413d88e86a1f92a40b93'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.793764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bd47edbc-30c8-49ef-8477-4d246dc79aca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a28a3ab2b7a413d88e86a1f92a40b93'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:57.814681 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1306/2657] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:57 2022] DELETE /identity/v3/users/8a28a3ab2b7a413d88e86a1f92a40b93 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:58.465513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:58.470744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:58.505836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec2a45c-6401-495e-b3fb-b0703813ff93 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:58.507472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:58.508323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:58.508323 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:58.547687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] MFA Rules not processed for user `471299b556bd4c22b51defc14b49d8eb`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:00:58.548944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:58.612485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:58.612485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:58.612485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:58.612485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] PATH_INFO: `/v3/projects/619278d625a947efabbd53f60ce6628f` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:58.612851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] RBAC: Authorizing `identity:delete_project(project_id=619278d625a947efabbd53f60ce6628f)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:58.615380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:58.664296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1307/2658] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:00:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:00:58.667190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:58.669525 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:58.683353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeadbee-f582-4c5d-9136-0ff7be985f91 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:58.688038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:58.688645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:58.688920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:58.689187 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:58.689978 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:58.691964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:58.737471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '619278d625a947efabbd53f60ce6628f', 'request_id': 'req-b87a9109-0b79-408c-b7b3-30b2c7c9510f'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:58.748397 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '619278d625a947efabbd53f60ce6628f', 'request_id': 'req-b87a9109-0b79-408c-b7b3-30b2c7c9510f'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:58.789651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1308/2659] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:00:58 2022] POST /identity/v3/projects => generated 377 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:58.793545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:58.806769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '619278d625a947efabbd53f60ce6628f', 'request_id': 'req-b87a9109-0b79-408c-b7b3-30b2c7c9510f'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:58.818122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] The token cache is being invalidate because project 619278d625a947efabbd53f60ce6628f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:00:58.818439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:58.819088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:58.820595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1352/2660] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:58 2022] DELETE /identity/v3/projects/619278d625a947efabbd53f60ce6628f => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:58.824784 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:58.828947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:58.830137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:58.862424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed2d3d-216e-41d0-86ee-2b0dc6f76993 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:58.867579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:58.868294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:58.868638 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:58.869004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:58.869654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:58.871615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:58.911511 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87a9109-0b79-408c-b7b3-30b2c7c9510f admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:58.916012 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:58.916658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:58.916920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:58.917191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] PATH_INFO: `/v3/users/efd194163aa9419e9014ef200d613fbc` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:58.918390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=efd194163aa9419e9014ef200d613fbc)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:58.920401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:59.040163 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1309/2661] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:00:58 2022] POST /identity/v3/users => generated 295 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:00:59.043961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:59.045607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:59.057687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3920b5-270f-4d20-90dc-dff7b0b536a4 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:59.062846 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:59.063429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:59.063708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:59.063981 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:59.064746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:59.066320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:59.075431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:00:59.081153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1310/2662] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:59 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:00:59.093060 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:59.102750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3ace47-6bdd-4ce0-8136-cbb7034b8302 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:59.107375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:59.108002 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:59.108278 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:59.108549 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] PATH_INFO: `/v3/projects/33c49c7ca4e44d45b2416ab67784f0eb/users/74bbf6ce6700410bafd4510241993345/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:59.125229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33c49c7ca4e44d45b2416ab67784f0eb, user_id=74bbf6ce6700410bafd4510241993345, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:59.127979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:59.186803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efd194163aa9419e9014ef200d613fbc'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:59.195500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efd194163aa9419e9014ef200d613fbc'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:59.196172 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efd194163aa9419e9014ef200d613fbc'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:59.197021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efd194163aa9419e9014ef200d613fbc'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:59.203873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1311/2663] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:59 2022] PUT /identity/v3/projects/33c49c7ca4e44d45b2416ab67784f0eb/users/74bbf6ce6700410bafd4510241993345/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:59.208050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:59.209966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:00:59.281583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd08fab-f629-44b0-8d30-6b1af3cbd919 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:59.286973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:59.287586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:59.287863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:59.288134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:59.288780 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:59.290371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:59.298595 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efd194163aa9419e9014ef200d613fbc'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:00:59.302433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1312/2664] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:00:59 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:00:59.306790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:59.318483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85279b7-71b1-4b04-aac4-af4d4aec0bbd admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:00:59.323724 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce83664-aa2b-441f-b3f4-1048ff793a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:00:59.324401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce83664-aa2b-441f-b3f4-1048ff793a8b admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:00:59.324676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce83664-aa2b-441f-b3f4-1048ff793a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:00:59.324945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce83664-aa2b-441f-b3f4-1048ff793a8b admin admin] PATH_INFO: `/v3/projects/33c49c7ca4e44d45b2416ab67784f0eb/users/74bbf6ce6700410bafd4510241993345/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:00:59.328149 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce83664-aa2b-441f-b3f4-1048ff793a8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=33c49c7ca4e44d45b2416ab67784f0eb, user_id=74bbf6ce6700410bafd4510241993345, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:00:59.330793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce83664-aa2b-441f-b3f4-1048ff793a8b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:00:59.334729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1353/2665] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:00:58 2022] DELETE /identity/v3/users/efd194163aa9419e9014ef200d613fbc => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:00:59.347364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1313/2666] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:00:59 2022] PUT /identity/v3/projects/33c49c7ca4e44d45b2416ab67784f0eb/users/74bbf6ce6700410bafd4510241993345/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:00:59.931687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:00:59.937531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:00.041050 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c795ca75-c537-4fb6-94cf-a2748d850d72 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:00.100994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:00.101640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:00.101925 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:00.102200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] PATH_INFO: `/v3/projects/8f5e959e140c421883424057d7631d89` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:00.103499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f5e959e140c421883424057d7631d89)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:00.105410 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:00.223100 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f5e959e140c421883424057d7631d89', 'request_id': 'req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:00.230424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f5e959e140c421883424057d7631d89', 'request_id': 'req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:00.233175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:00.382248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f5e959e140c421883424057d7631d89', 'request_id': 'req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:00.402244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] The token cache is being invalidate because project 8f5e959e140c421883424057d7631d89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:00.402591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:00.403251 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c090f449-f0f7-4a5d-8b9d-d4f06ee6caf7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:00.406515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1354/2667] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:00:59 2022] DELETE /identity/v3/projects/8f5e959e140c421883424057d7631d89 => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:00.412184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33434490-1bba-45f7-a7ef-45ef9def04a5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:00.416583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33434490-1bba-45f7-a7ef-45ef9def04a5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:00.416583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33434490-1bba-45f7-a7ef-45ef9def04a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:00.439357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-33434490-1bba-45f7-a7ef-45ef9def04a5 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:00.439357 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-33434490-1bba-45f7-a7ef-45ef9def04a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:00.538883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1314/2668] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:00.547352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae105b55-a652-4040-8df0-a20952de440f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:00.547702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae105b55-a652-4040-8df0-a20952de440f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:00.547980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae105b55-a652-4040-8df0-a20952de440f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:00.584516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-ae105b55-a652-4040-8df0-a20952de440f None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:00.587028 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ae105b55-a652-4040-8df0-a20952de440f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:00.603058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1355/2669] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:00.607264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33434490-1bba-45f7-a7ef-45ef9def04a5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:00.617536 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33434490-1bba-45f7-a7ef-45ef9def04a5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:00.624977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:00.625600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:00.625880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:00.626152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:00.626882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:00.627213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:00.629025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:00.633586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:00.644615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1315/2670] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:00.648986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae105b55-a652-4040-8df0-a20952de440f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:00.658554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae105b55-a652-4040-8df0-a20952de440f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:00.663823 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:00.664448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:00.664723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:00.664991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:00.665636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:00.667598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:00.731637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1356/2671] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:00 2022] POST /identity/v3/projects => generated 367 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:00.733390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:00.746003 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80261373-c18d-4f2f-8fc5-4eec6e604511 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:00.836129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:00.836543 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:00.836822 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:00.837092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:00.838068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:00.840229 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:01.000091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1316/2672] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:00 2022] POST /identity/v3/users => generated 284 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:01.001445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:01.012977 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9e7844-219c-498a-85d0-22711f89f352 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:01.020347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:01.021224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.021696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.022061 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.022844 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:01.024666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:01.173101 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1357/2673] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:01 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:01.174269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:01.184649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fc3aab-ee2c-4d2c-8ec9-180132e5b692 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:01.190361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2111ec09-271c-4a07-91de-fd594e9407c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:01.191110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2111ec09-271c-4a07-91de-fd594e9407c0 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.191484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2111ec09-271c-4a07-91de-fd594e9407c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.191852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2111ec09-271c-4a07-91de-fd594e9407c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.192589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2111ec09-271c-4a07-91de-fd594e9407c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:01.194362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2111ec09-271c-4a07-91de-fd594e9407c0 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:01.206168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1317/2674] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:01.212531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:01.221471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558c626a-8347-4fe9-ba06-3cd50b67abf0 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:01.227593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:01.228569 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.229066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.229656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] PATH_INFO: `/v3/projects/f92727dac2cd41e8affc40fd57c0bfd5/users/49af3a4b837e44e28e54f312287df11b/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.246146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f92727dac2cd41e8affc40fd57c0bfd5, user_id=49af3a4b837e44e28e54f312287df11b, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:01.249422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:01.334559 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1358/2675] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:01 2022] PUT /identity/v3/projects/f92727dac2cd41e8affc40fd57c0bfd5/users/49af3a4b837e44e28e54f312287df11b/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:01.349086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d22e40-4684-461f-b78a-d595dc0aaa34 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.349367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d22e40-4684-461f-b78a-d595dc0aaa34 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.349633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d22e40-4684-461f-b78a-d595dc0aaa34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.383498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-e3d22e40-4684-461f-b78a-d595dc0aaa34 None None] MFA Rules not processed for user `49af3a4b837e44e28e54f312287df11b`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:01.384788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-e3d22e40-4684-461f-b78a-d595dc0aaa34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:01.472806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1318/2676] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:01 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:01.477232 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:01.566504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c05ec4-dfd4-467a-af80-209b97d32d54 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:01.572081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:01.572709 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.573369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.573369 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.574320 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:01.576121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:01.659256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1359/2677] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:01 2022] POST /identity/v3/projects => generated 363 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:01.663174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d22e40-4684-461f-b78a-d595dc0aaa34 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:01.676626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d22e40-4684-461f-b78a-d595dc0aaa34 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:01.762203 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:01.763716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.764586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.765038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.766321 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:01.769560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:01.782072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:01.866673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1319/2678] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:01 2022] POST /identity/v3/users => generated 288 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:01.867839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:01.879160 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d4298-fe45-4c71-ae48-2ccad433dfb4 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:01.885548 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:01.886698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.887360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.887643 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.888986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:01.892648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:01.932288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1360/2679] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:01 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:01:01.934545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:01.946429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa64aef9-ca44-4692-bdd1-3c50d5977f98 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:01.952004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:01.952907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:01.953498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:01.953771 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] PATH_INFO: `/v3/projects/157eb2614c014539b435e9f17b4edbf6/users/226f6f15bb4644948f5b1302d1fcb293/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:01.991063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=157eb2614c014539b435e9f17b4edbf6, user_id=226f6f15bb4644948f5b1302d1fcb293, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:02.003971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:02.126898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1320/2680] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:01 2022] PUT /identity/v3/projects/157eb2614c014539b435e9f17b4edbf6/users/226f6f15bb4644948f5b1302d1fcb293/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:02.131380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:02.145076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b60913-d99f-4c28-a8d1-9c769bb46cc1 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:02.246620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:02.264058 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374ce968-fdd2-4809-a138-9fe0fc36ac96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:02.264949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ce968-fdd2-4809-a138-9fe0fc36ac96 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:02.265297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ce968-fdd2-4809-a138-9fe0fc36ac96 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:02.265636 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ce968-fdd2-4809-a138-9fe0fc36ac96 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:02.266463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374ce968-fdd2-4809-a138-9fe0fc36ac96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:02.274512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374ce968-fdd2-4809-a138-9fe0fc36ac96 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:02.283543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1361/2681] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:02 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:02.287816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:02.308723 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d195053a-80cb-4a6e-85e5-f362ca39c8e0 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:02.336390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:02.337664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:02.344364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:02.344883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] PATH_INFO: `/v3/projects/157eb2614c014539b435e9f17b4edbf6/users/226f6f15bb4644948f5b1302d1fcb293/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:02.348721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] RBAC: Authorizing `identity:create_grant(project_id=157eb2614c014539b435e9f17b4edbf6, user_id=226f6f15bb4644948f5b1302d1fcb293, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:02.353477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:02.382453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1321/2682] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:02 2022] PUT /identity/v3/projects/157eb2614c014539b435e9f17b4edbf6/users/226f6f15bb4644948f5b1302d1fcb293/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:02.392388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583df970-8b6d-401d-96a8-0e04b6c2689f None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:02.395264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583df970-8b6d-401d-96a8-0e04b6c2689f None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:02.396351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583df970-8b6d-401d-96a8-0e04b6c2689f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:02.446448 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-583df970-8b6d-401d-96a8-0e04b6c2689f None None] MFA Rules not processed for user `226f6f15bb4644948f5b1302d1fcb293`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:02.448134 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-583df970-8b6d-401d-96a8-0e04b6c2689f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:02.597571 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1362/2683] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:02 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:02.609096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:02.618929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:02.725166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159c63c9-e375-44d6-bc33-a48b9c64d664 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:02.737929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee4282e-e040-4033-b457-9b6329716bfb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:02.744302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee4282e-e040-4033-b457-9b6329716bfb service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:02.744302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee4282e-e040-4033-b457-9b6329716bfb service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:02.744302 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee4282e-e040-4033-b457-9b6329716bfb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:02.759221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee4282e-e040-4033-b457-9b6329716bfb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:02.761097 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee4282e-e040-4033-b457-9b6329716bfb service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:02.775029 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1322/2684] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:02 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:03.071395 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583df970-8b6d-401d-96a8-0e04b6c2689f None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:03.091905 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583df970-8b6d-401d-96a8-0e04b6c2689f None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:03.108634 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:03.110279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:03.111308 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:03.112113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:03.220365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:03.222020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:03.233869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1363/2685] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:03 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:06.380000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:06.380492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:06.380606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:06.403082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:06.404706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:06.419036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] MFA Rules not processed for user `74bbf6ce6700410bafd4510241993345`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:06.420949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:06.491906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4e09d0-8bdd-484d-891c-43026446e962 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:06.498154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:06.499929 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:06.504231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:06.504231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] PATH_INFO: `/v3/users/c682afa0ab1f401f850514e5baf7633a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:06.504231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c682afa0ab1f401f850514e5baf7633a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:06.504231 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:06.505964 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1323/2686] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:06 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:06.520841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:06.521344 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:06.523317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:06.575656 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:06.624301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251c9ba6-e752-4495-85d5-486c00d031e0 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:06.630101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97f847-0290-426c-bf3b-a76d83b6ea11 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:06.630754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97f847-0290-426c-bf3b-a76d83b6ea11 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:06.631032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97f847-0290-426c-bf3b-a76d83b6ea11 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:06.631300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97f847-0290-426c-bf3b-a76d83b6ea11 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:06.633245 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0f97f847-0290-426c-bf3b-a76d83b6ea11 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:06.701025 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97f847-0290-426c-bf3b-a76d83b6ea11 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:06.702538 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97f847-0290-426c-bf3b-a76d83b6ea11 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:06.705789 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c682afa0ab1f401f850514e5baf7633a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:06.715176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1324/2687] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:06 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:06.716968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c682afa0ab1f401f850514e5baf7633a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:06.718009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c682afa0ab1f401f850514e5baf7633a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:06.719036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c682afa0ab1f401f850514e5baf7633a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:06.793888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c682afa0ab1f401f850514e5baf7633a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:06.812524 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1364/2688] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:06 2022] DELETE /identity/v3/users/c682afa0ab1f401f850514e5baf7633a => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:06.818051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:06.818448 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:06.818917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:06.857454 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:06.859004 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:06.931718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1325/2689] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:06.945325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:06.947401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:07.079527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c24ef5d-7dbd-4365-a89b-74e3b6b258f8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:07.079527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:07.079527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:07.079527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:07.079995 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:07.090816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:07.092863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:07.103498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1365/2690] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:07.662326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:07.663885 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:07.675945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a69c8-80c7-4d0a-9526-8864caca71da None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:07.687648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:07.687648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:07.687648 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:07.688200 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] PATH_INFO: `/v3/projects/bd948607d91d4689b7b169a075aac661` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:07.689283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd948607d91d4689b7b169a075aac661)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:07.691358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:07.821953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd948607d91d4689b7b169a075aac661', 'request_id': 'req-9256173a-1bd4-498c-b404-1402d90b7daa'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:07.890622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:07.892133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:07.896654 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd948607d91d4689b7b169a075aac661', 'request_id': 'req-9256173a-1bd4-498c-b404-1402d90b7daa'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:07.898044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:07.906382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd948607d91d4689b7b169a075aac661', 'request_id': 'req-9256173a-1bd4-498c-b404-1402d90b7daa'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:07.906549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4483d5-db75-4d39-ab26-bd9aefa5c73e service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:07.917051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] The token cache is being invalidate because project bd948607d91d4689b7b169a075aac661 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:07.917343 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:07.918055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9256173a-1bd4-498c-b404-1402d90b7daa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:07.919606 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1326/2691] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:07 2022] DELETE /identity/v3/projects/bd948607d91d4689b7b169a075aac661 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:07.924370 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c3580-8d0d-4e9d-b0b7-133a0e36bf2d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:07.924733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c3580-8d0d-4e9d-b0b7-133a0e36bf2d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:07.925010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c3580-8d0d-4e9d-b0b7-133a0e36bf2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:07.945058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-d44c3580-8d0d-4e9d-b0b7-133a0e36bf2d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:07.946266 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d44c3580-8d0d-4e9d-b0b7-133a0e36bf2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:07.969208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:07.969855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:07.970131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:07.970429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] PATH_INFO: `/v3/users/b2c2db836b2b4a0cab635d2546950b83` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:07.971778 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2c2db836b2b4a0cab635d2546950b83)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:07.973869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.032778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1327/2692] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:08.040643 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e5e86c-86cc-41f4-9e96-6663571413b1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.040938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e5e86c-86cc-41f4-9e96-6663571413b1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.041217 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e5e86c-86cc-41f4-9e96-6663571413b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.065456 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-60e5e86c-86cc-41f4-9e96-6663571413b1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:08.066744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-60e5e86c-86cc-41f4-9e96-6663571413b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:08.094461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1328/2693] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:08.094461 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e5e86c-86cc-41f4-9e96-6663571413b1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.095310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e5e86c-86cc-41f4-9e96-6663571413b1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:08.101727 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:08.102372 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.102762 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.102955 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.103660 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:08.104144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:08.105749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.121327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1329/2694] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:08.130201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.143273 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab5014d-b4a1-4287-a558-8520941bdf3c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:08.149529 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:08.150146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.150548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.151292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.151710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:08.153422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.220989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:08.229618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2c2db836b2b4a0cab635d2546950b83'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:08.256937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2c2db836b2b4a0cab635d2546950b83'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:08.259092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2c2db836b2b4a0cab635d2546950b83'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:08.260533 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2c2db836b2b4a0cab635d2546950b83'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:08.290878 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1330/2695] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:08 2022] POST /identity/v3/projects => generated 391 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:08.292246 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.295148 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:08.366986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-84743fe4-6c70-4a09-a551-fb1ed160832a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2c2db836b2b4a0cab635d2546950b83'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:08.380020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb9afe1-9c56-490e-97f9-f16afe10fc36 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:08.384020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:08.384639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.384912 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.385174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.385814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:08.389698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.406131 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1366/2696] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:07 2022] DELETE /identity/v3/users/b2c2db836b2b4a0cab635d2546950b83 => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:08.412205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.412385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.412635 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.442233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:08.443517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:08.517261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1367/2697] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:08.526908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1331/2698] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:08 2022] POST /identity/v3/users => generated 296 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:08.528003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.530164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:08.542690 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.563629 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f554db3-4743-486e-aa3e-cd49e1c968e8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:08.577961 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:08.578843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.579401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.579773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.580568 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:08.583078 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.603810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1332/2699] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:08 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:08.604960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.618731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109a099-be2c-47ec-9604-123d63632a1e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:08.625349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:08.626083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.626609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.626850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14457337-5fa4-4a01-980f-670ef6032b18 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:08.627663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.627776 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:08.629307 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.631297 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10be2c6-50f5-4132-983f-47059676134f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:08.632089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10be2c6-50f5-4132-983f-47059676134f service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.632466 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10be2c6-50f5-4132-983f-47059676134f service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.632837 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10be2c6-50f5-4132-983f-47059676134f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.641599 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1333/2700] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:08 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:08.644477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10be2c6-50f5-4132-983f-47059676134f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:08.646057 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10be2c6-50f5-4132-983f-47059676134f service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.647173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.661368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1368/2701] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:08.675742 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc2adf7-22aa-421b-bf62-dd85f61589f1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:08.682966 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:08.683655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.683969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.684247 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] PATH_INFO: `/v3/projects/b5f9accd3435416d9749ccab9b4e4d9d/users/fd98ae518f0d4956a0c08ca8caaeebac/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.695069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:08.708389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5f9accd3435416d9749ccab9b4e4d9d, user_id=fd98ae518f0d4956a0c08ca8caaeebac, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:08.710398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:08.796876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1334/2702] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:08 2022] PUT /identity/v3/projects/b5f9accd3435416d9749ccab9b4e4d9d/users/fd98ae518f0d4956a0c08ca8caaeebac/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:08.808260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:08.808551 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:08.808842 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:08.840593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] MFA Rules not processed for user `fd98ae518f0d4956a0c08ca8caaeebac`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:08.842357 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:08.905380 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:08.937762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1369/2703] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:08 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:08.953313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:08.955284 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:09.049908 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718f3a16-d3f9-48e0-8078-f116ef485df7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:09.054081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:09.059515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:09.059515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:09.059515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:09.072319 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:09.073664 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:09.093702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1335/2704] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:08 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:09.310963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:09.314326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:09.454540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d61af68-b6a3-46bc-b628-f23752aa1840 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:09.460685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:09.461484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:09.461767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:09.462040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] PATH_INFO: `/v3/projects/13e79bfb82b84cd693ccb7faa3c0c668` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:09.463257 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=13e79bfb82b84cd693ccb7faa3c0c668)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:09.465506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:09.615903 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '13e79bfb82b84cd693ccb7faa3c0c668', 'request_id': 'req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:09.622445 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '13e79bfb82b84cd693ccb7faa3c0c668', 'request_id': 'req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:09.721899 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '13e79bfb82b84cd693ccb7faa3c0c668', 'request_id': 'req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:09.735673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] The token cache is being invalidate because project 13e79bfb82b84cd693ccb7faa3c0c668 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:09.735958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:09.736670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:09.738154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1370/2705] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:09 2022] DELETE /identity/v3/projects/13e79bfb82b84cd693ccb7faa3c0c668 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:09.742256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:09.754898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa06b64-d70f-40f4-a89a-f380fbab1098 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:09.821367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:09.821929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:09.822208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:09.822531 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] PATH_INFO: `/v3/users/77f86bd2face4ef18d1319960271e953` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:09.823686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] RBAC: Authorizing `identity:delete_user(user_id=77f86bd2face4ef18d1319960271e953)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:09.825645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:10.073634 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77f86bd2face4ef18d1319960271e953'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:10.082530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77f86bd2face4ef18d1319960271e953'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:10.083176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77f86bd2face4ef18d1319960271e953'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:10.083857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77f86bd2face4ef18d1319960271e953'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:10.165136 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77f86bd2face4ef18d1319960271e953'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:10.191253 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1336/2706] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:09 2022] DELETE /identity/v3/users/77f86bd2face4ef18d1319960271e953 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:10.705044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:10.707233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:10.731682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6d693c-f9b2-4280-8e0c-8b337c0b21fd admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:10.815313 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:10.815941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:10.816234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:10.816486 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] PATH_INFO: `/v3/projects/30c05d9f04374159abfe30618575a755` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:10.817613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=30c05d9f04374159abfe30618575a755)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:10.819529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:10.840463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:10.882557 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:10.884083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:10.897555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f456e18-bd47-45c6-9e6d-81fcc8041fdd admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:10.902049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:10.902708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:10.902984 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:10.903262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] PATH_INFO: `/v3/users/49af3a4b837e44e28e54f312287df11b` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:10.905192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=49af3a4b837e44e28e54f312287df11b)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:10.907114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:10.909415 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:10.983413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30c05d9f04374159abfe30618575a755', 'request_id': 'req-0f749fba-8e31-4484-ab65-2fc35408fae9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:10.989794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30c05d9f04374159abfe30618575a755', 'request_id': 'req-0f749fba-8e31-4484-ab65-2fc35408fae9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.104959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30c05d9f04374159abfe30618575a755', 'request_id': 'req-0f749fba-8e31-4484-ab65-2fc35408fae9'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.120403 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] The token cache is being invalidate because project 30c05d9f04374159abfe30618575a755 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:11.120698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.121365 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0f749fba-8e31-4484-ab65-2fc35408fae9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.123708 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1371/2707] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:10 2022] DELETE /identity/v3/projects/30c05d9f04374159abfe30618575a755 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:11.128451 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33829f0-9591-48f4-a437-978f25732897 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.128739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33829f0-9591-48f4-a437-978f25732897 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.129013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33829f0-9591-48f4-a437-978f25732897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.149728 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49af3a4b837e44e28e54f312287df11b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.152421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c33829f0-9591-48f4-a437-978f25732897 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:11.153641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c33829f0-9591-48f4-a437-978f25732897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:11.158906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49af3a4b837e44e28e54f312287df11b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.161665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49af3a4b837e44e28e54f312287df11b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.162517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49af3a4b837e44e28e54f312287df11b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.214675 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1372/2708] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:11.222698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c311959-0ea6-4686-8f54-feaf297926d0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.222989 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c311959-0ea6-4686-8f54-feaf297926d0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.223264 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c311959-0ea6-4686-8f54-feaf297926d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.241027 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-891187cf-65dd-4d46-9f3c-972693e6d6c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49af3a4b837e44e28e54f312287df11b'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:11.242515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-1c311959-0ea6-4686-8f54-feaf297926d0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:11.243673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1c311959-0ea6-4686-8f54-feaf297926d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:11.254723 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1373/2709] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:11.257191 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c311959-0ea6-4686-8f54-feaf297926d0 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.263898 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1337/2710] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:10 2022] DELETE /identity/v3/users/49af3a4b837e44e28e54f312287df11b => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:11.265429 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.265768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.266053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.270125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c311959-0ea6-4686-8f54-feaf297926d0 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.289603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:11.290899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:11.328045 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:11.328654 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.328927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.329194 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.329830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:11.330125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:11.331805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:11.343774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1374/2711] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 09:01:11.348106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.352381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1338/2712] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:11.358647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccb0e70-af38-400b-b08d-5f849f5af21b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.362623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.365688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:11.365941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:11.366039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.366039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.366039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.366039 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:11.367241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:11.432449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727536f3-00d2-49b4-85af-cf7c62c40a04 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.436839 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:11.437442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.437715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.437979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.446774 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:11.448180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:11.458176 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1339/2713] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:11.521586 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1375/2714] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:11 2022] POST /identity/v3/projects => generated 379 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:11.523502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.539125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a2b4ee-618f-4f04-86d1-86dd2da619c7 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.612264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:11.612879 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.613205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.613481 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.614101 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:11.616034 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:11.706445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1340/2715] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:11 2022] POST /identity/v3/users => generated 290 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:11.710359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.721813 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d9c9de-4c09-4020-8f16-c109ad5ae607 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.727641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:11.727963 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.728455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.728894 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.729710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:11.732175 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:11.744245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1376/2716] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:11.749670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.762663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f601d8-05c6-42b3-8b2e-010891e1fcb5 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.770374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:11.772476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.772476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.772476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.772476 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:11.774303 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:11.795178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1341/2717] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:01:11.796513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.808535 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd6137f-a36b-433a-b924-1a0233e4751a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.816810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298e07f3-5de7-4e8d-a472-311dfb299510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:11.817516 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298e07f3-5de7-4e8d-a472-311dfb299510 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.817794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298e07f3-5de7-4e8d-a472-311dfb299510 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.819541 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298e07f3-5de7-4e8d-a472-311dfb299510 admin admin] PATH_INFO: `/v3/projects/b7ba426889f548129d1c9ad246051754/users/152a31d816a948279d974feec2a79e52/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.836370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298e07f3-5de7-4e8d-a472-311dfb299510 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7ba426889f548129d1c9ad246051754, user_id=152a31d816a948279d974feec2a79e52, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:11.840018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298e07f3-5de7-4e8d-a472-311dfb299510 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:11.929201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:11.933122 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:11.946670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1377/2718] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:11 2022] PUT /identity/v3/projects/b7ba426889f548129d1c9ad246051754/users/152a31d816a948279d974feec2a79e52/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:11.947686 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff5d320-088b-40f9-b7bc-ff0879e2315b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:11.951757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99053911-8c54-44ac-892e-bca66832ebbc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:11.952116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99053911-8c54-44ac-892e-bca66832ebbc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:11.952375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99053911-8c54-44ac-892e-bca66832ebbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:11.980743 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-99053911-8c54-44ac-892e-bca66832ebbc None None] MFA Rules not processed for user `152a31d816a948279d974feec2a79e52`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:11.982640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-99053911-8c54-44ac-892e-bca66832ebbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:12.018524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.019342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.019694 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.020043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] PATH_INFO: `/v3/projects/f92727dac2cd41e8affc40fd57c0bfd5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.021424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=f92727dac2cd41e8affc40fd57c0bfd5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.023363 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.061042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1378/2719] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:11 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:12.065602 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99053911-8c54-44ac-892e-bca66832ebbc None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.075700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99053911-8c54-44ac-892e-bca66832ebbc None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.080447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.081033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.083656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.083656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.083656 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.084118 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.159994 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f92727dac2cd41e8affc40fd57c0bfd5', 'request_id': 'req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.183284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1379/2720] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:12 2022] POST /identity/v3/projects => generated 381 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:12.187208 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.245412 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f92727dac2cd41e8affc40fd57c0bfd5', 'request_id': 'req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.249084 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7516674-0821-4c49-bbff-c98e0fbd13b0 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.253913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f92727dac2cd41e8affc40fd57c0bfd5', 'request_id': 'req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.254909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.255768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.256014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.256306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.257043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.259545 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.266198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] The token cache is being invalidate because project f92727dac2cd41e8affc40fd57c0bfd5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:12.266607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.267334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.269479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1342/2721] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:11 2022] DELETE /identity/v3/projects/f92727dac2cd41e8affc40fd57c0bfd5 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:12.273554 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.275085 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:12.280589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:12.342972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd624e2d-ccc1-4fa3-851b-bce8ef2430ee admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.347585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.348222 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.348789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.349105 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] PATH_INFO: `/v3/users/226f6f15bb4644948f5b1302d1fcb293` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.350252 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] RBAC: Authorizing `identity:delete_user(user_id=226f6f15bb4644948f5b1302d1fcb293)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.352206 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.357042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1380/2722] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:12 2022] POST /identity/v3/users => generated 297 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:12.361179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.362820 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:12.390156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce09e196-83ec-4ea3-8923-e7af378af081 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.395404 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.396115 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.396477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.396771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.397491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.399299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.412367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1381/2723] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:12 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:12.416258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.426774 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39ccc1-b5f1-4c6f-974d-a354d4c9196e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.431821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.432540 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.432839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.433364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] PATH_INFO: `/v3/projects/543deab5257944d8ad67b5fe5cf3fbf0/users/0d8d27c70f9d4697bbde2eb1cd2b5344/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.448538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=543deab5257944d8ad67b5fe5cf3fbf0, user_id=0d8d27c70f9d4697bbde2eb1cd2b5344, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.452531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.455140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:12.539399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1382/2724] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:12 2022] PUT /identity/v3/projects/543deab5257944d8ad67b5fe5cf3fbf0/users/0d8d27c70f9d4697bbde2eb1cd2b5344/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:12.543266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.551092 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '226f6f15bb4644948f5b1302d1fcb293'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.558960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '226f6f15bb4644948f5b1302d1fcb293'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.560234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '226f6f15bb4644948f5b1302d1fcb293'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.561010 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '226f6f15bb4644948f5b1302d1fcb293'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.607205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1334bfff-320d-4744-a652-4601c0731b54 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.612331 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.613013 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.613139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-bf2d1ef5-c9a7-4672-9691-ca373b968781 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '226f6f15bb4644948f5b1302d1fcb293'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:12.613243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.613616 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.614373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.616078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.633201 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1383/2725] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:12 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:01:12.637736 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.640633 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:12.647068 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1343/2726] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:12 2022] DELETE /identity/v3/users/226f6f15bb4644948f5b1302d1fcb293 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:12.654037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2391b958-ecee-4696-83b8-5341b606882c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.711040 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.711809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.712166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.712515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] PATH_INFO: `/v3/projects/543deab5257944d8ad67b5fe5cf3fbf0/users/0d8d27c70f9d4697bbde2eb1cd2b5344/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.715375 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] RBAC: Authorizing `identity:create_grant(project_id=543deab5257944d8ad67b5fe5cf3fbf0, user_id=0d8d27c70f9d4697bbde2eb1cd2b5344, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.718124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.729179 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1384/2727] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:12 2022] PUT /identity/v3/projects/543deab5257944d8ad67b5fe5cf3fbf0/users/0d8d27c70f9d4697bbde2eb1cd2b5344/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:12.739116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.739400 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.739672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.769089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] MFA Rules not processed for user `0d8d27c70f9d4697bbde2eb1cd2b5344`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:12.770330 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:12.855578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1344/2728] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:12 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:12.875720 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:12.878066 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:12.937712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac415d-492b-4be6-ab11-4b4ab9800452 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:12.945467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:12.945467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:12.945467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:12.945467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:12.955151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:12.956584 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:12.967566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:12.973338 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1385/2729] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:12 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:12.990055 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:13.100797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.102446 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:13.182563 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecbbd44-5e09-4663-b172-ece92d883cd8 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.187745 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.191198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.191198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.191198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] PATH_INFO: `/v3/projects/157eb2614c014539b435e9f17b4edbf6` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.191198 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=157eb2614c014539b435e9f17b4edbf6)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.191938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:13.230997 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.240116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70740fb-b8a8-43d1-a283-1a08110a3e9e service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.244254 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bdccb7-41d1-430e-945f-404b19e0a56d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.244839 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bdccb7-41d1-430e-945f-404b19e0a56d service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.245112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bdccb7-41d1-430e-945f-404b19e0a56d service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.245381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bdccb7-41d1-430e-945f-404b19e0a56d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.247117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-74bdccb7-41d1-430e-945f-404b19e0a56d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:13.277887 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:13.310171 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:13.313645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '157eb2614c014539b435e9f17b4edbf6', 'request_id': 'req-28e2d569-e783-4523-983a-6d8dfe892b52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:13.321514 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bdccb7-41d1-430e-945f-404b19e0a56d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.322986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bdccb7-41d1-430e-945f-404b19e0a56d service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:13.332429 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1386/2730] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:13 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:13.379427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '157eb2614c014539b435e9f17b4edbf6', 'request_id': 'req-28e2d569-e783-4523-983a-6d8dfe892b52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:13.386295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '157eb2614c014539b435e9f17b4edbf6', 'request_id': 'req-28e2d569-e783-4523-983a-6d8dfe892b52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:13.398032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] The token cache is being invalidate because project 157eb2614c014539b435e9f17b4edbf6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:13.398341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:13.398996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-28e2d569-e783-4523-983a-6d8dfe892b52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:13.400585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1345/2731] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:13 2022] DELETE /identity/v3/projects/157eb2614c014539b435e9f17b4edbf6 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:13.406242 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56d875-2570-4bca-b42b-fc0bba55792b None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.406597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56d875-2570-4bca-b42b-fc0bba55792b None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.406870 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56d875-2570-4bca-b42b-fc0bba55792b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.427564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-be56d875-2570-4bca-b42b-fc0bba55792b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:13.428733 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-be56d875-2570-4bca-b42b-fc0bba55792b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:13.487930 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1387/2732] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:13.496071 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb7923-b5d7-4349-869d-f6dc97b85444 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.496375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb7923-b5d7-4349-869d-f6dc97b85444 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.496715 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb7923-b5d7-4349-869d-f6dc97b85444 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.518095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-3fcb7923-b5d7-4349-869d-f6dc97b85444 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:13.519423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-3fcb7923-b5d7-4349-869d-f6dc97b85444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:13.530142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1346/2733] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:13.534505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56d875-2570-4bca-b42b-fc0bba55792b None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.544804 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56d875-2570-4bca-b42b-fc0bba55792b None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.549553 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.550144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.550449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.550729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.554152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:13.554152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.554152 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:13.565262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1388/2734] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:13.569673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb7923-b5d7-4349-869d-f6dc97b85444 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.579318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb7923-b5d7-4349-869d-f6dc97b85444 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.584282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.584882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.585157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.585487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.586035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.587991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:13.666313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1347/2735] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:13 2022] POST /identity/v3/projects => generated 387 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:13.671074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.681224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0627ff63-f79a-4e7a-89c2-e216159d3baf admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.764980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.765640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.765947 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.766222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.767014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.771413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:13.847770 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1389/2736] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:13 2022] POST /identity/v3/users => generated 294 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:13.851993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.862637 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02230279-ff1f-4ef3-ab8f-5e06aa477de1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.867213 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.867811 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.868088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.868364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.869058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.870688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:13.880470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1348/2737] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:13.884428 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.895567 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10658e81-5048-4666-bf57-1c9199aa1c8b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.900364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb371c3-6d11-4c81-b869-d05c430383d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.900993 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb371c3-6d11-4c81-b869-d05c430383d4 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.901268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb371c3-6d11-4c81-b869-d05c430383d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.901537 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb371c3-6d11-4c81-b869-d05c430383d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.902166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb371c3-6d11-4c81-b869-d05c430383d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.904362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb371c3-6d11-4c81-b869-d05c430383d4 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:13.914931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1390/2738] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:13.919140 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:13.929254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4910a2-95ca-486e-9ef4-6655c553b4da admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:13.934117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:13.934824 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:13.935103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:13.935382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] PATH_INFO: `/v3/projects/2e972a6da88c4def8c30a434f31fb2cf/users/a06fe82be7014cdc80b396b2a0a0feab/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:13.950152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e972a6da88c4def8c30a434f31fb2cf, user_id=a06fe82be7014cdc80b396b2a0a0feab, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:13.953587 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:14.024050 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1349/2739] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:13 2022] PUT /identity/v3/projects/2e972a6da88c4def8c30a434f31fb2cf/users/a06fe82be7014cdc80b396b2a0a0feab/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:14.092803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe98be-2efc-42e1-ac46-32a52e9af727 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.093086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe98be-2efc-42e1-ac46-32a52e9af727 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.093358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe98be-2efc-42e1-ac46-32a52e9af727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.121594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7bbe98be-2efc-42e1-ac46-32a52e9af727 None None] MFA Rules not processed for user `a06fe82be7014cdc80b396b2a0a0feab`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:14.124088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7bbe98be-2efc-42e1-ac46-32a52e9af727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:14.151347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:14.190151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1391/2740] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:14 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:14.194589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:14.277361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f4a34-9a9d-4ca9-9bf9-f242b49c2fc1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:14.282541 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:14.283162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.283468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.283741 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.284387 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:14.286286 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:14.296502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:14.406256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1350/2741] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:14 2022] POST /identity/v3/projects => generated 385 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:14.410544 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbe98be-2efc-42e1-ac46-32a52e9af727 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:14.422120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbe98be-2efc-42e1-ac46-32a52e9af727 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:14.475959 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:14.476476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.476764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.477021 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.477645 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:14.479579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:14.598433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1392/2742] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:14 2022] POST /identity/v3/users => generated 299 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:14.602611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:14.622275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c6dac0-2cfe-4120-bf07-496c8fbe1f7c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:14.629910 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:14.630677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.631028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.631300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.632146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:14.634261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:14.645608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1351/2743] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:14.649865 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:14.660004 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e7b8b-8110-408e-8c80-2abc5bdd12f9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:14.664657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:14.665306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.665589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.665862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] PATH_INFO: `/v3/projects/f853535755d04bb6830733995dac67ae/users/74ff11f4a2364310a06888ab10d2e43e/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.682368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f853535755d04bb6830733995dac67ae, user_id=74ff11f4a2364310a06888ab10d2e43e, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:14.685795 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:14.761378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1393/2744] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:14 2022] PUT /identity/v3/projects/f853535755d04bb6830733995dac67ae/users/74ff11f4a2364310a06888ab10d2e43e/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:14.766075 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:14.776545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17281c7c-002c-4b8a-83c0-87f8324176fc admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:14.834575 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce675dd9-18b0-4092-b43a-d06fc08d0dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:14.835265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce675dd9-18b0-4092-b43a-d06fc08d0dca admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.835539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce675dd9-18b0-4092-b43a-d06fc08d0dca admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.835801 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce675dd9-18b0-4092-b43a-d06fc08d0dca admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.836430 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce675dd9-18b0-4092-b43a-d06fc08d0dca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:14.837990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce675dd9-18b0-4092-b43a-d06fc08d0dca admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:14.849058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1352/2745] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:14.853007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:14.865485 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24d2ddd-1ccf-4d17-8a0d-2760faae8b8a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:14.870333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:14.871023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.871298 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.871566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] PATH_INFO: `/v3/projects/f853535755d04bb6830733995dac67ae/users/74ff11f4a2364310a06888ab10d2e43e/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.873647 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] RBAC: Authorizing `identity:create_grant(project_id=f853535755d04bb6830733995dac67ae, user_id=74ff11f4a2364310a06888ab10d2e43e, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:14.876356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:14.888350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1394/2746] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:14 2022] PUT /identity/v3/projects/f853535755d04bb6830733995dac67ae/users/74ff11f4a2364310a06888ab10d2e43e/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:14.899733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03945a-ecf3-4690-a979-dae53b605065 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:14.900161 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03945a-ecf3-4690-a979-dae53b605065 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:14.900576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b03945a-ecf3-4690-a979-dae53b605065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:14.945056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-5b03945a-ecf3-4690-a979-dae53b605065 None None] MFA Rules not processed for user `74ff11f4a2364310a06888ab10d2e43e`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:14.946269 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5b03945a-ecf3-4690-a979-dae53b605065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:15.017819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1353/2747] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:14 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:15.029971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:15.031727 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:15.088349 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b31bf-18e1-49c3-9775-e144324124be admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:15.092622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e58c8f8-1dfa-4005-8828-7b011003c9ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:15.093223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58c8f8-1dfa-4005-8828-7b011003c9ff service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:15.095079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58c8f8-1dfa-4005-8828-7b011003c9ff service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:15.095079 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58c8f8-1dfa-4005-8828-7b011003c9ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:15.102901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e58c8f8-1dfa-4005-8828-7b011003c9ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:15.104444 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e58c8f8-1dfa-4005-8828-7b011003c9ff service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:15.116401 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1395/2748] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:15 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:16.297915 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b03945a-ecf3-4690-a979-dae53b605065 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:16.368951 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b03945a-ecf3-4690-a979-dae53b605065 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:16.373894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:16.374566 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:16.374851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:16.375119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] PATH_INFO: `/v3/users/a06fe82be7014cdc80b396b2a0a0feab` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:16.376257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a06fe82be7014cdc80b396b2a0a0feab)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:16.378174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:16.379853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:16.580758 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a06fe82be7014cdc80b396b2a0a0feab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:16.589945 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a06fe82be7014cdc80b396b2a0a0feab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:16.591401 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a06fe82be7014cdc80b396b2a0a0feab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:16.592864 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a06fe82be7014cdc80b396b2a0a0feab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:16.687816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a06fe82be7014cdc80b396b2a0a0feab'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:16.710747 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1354/2749] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:16 2022] DELETE /identity/v3/users/a06fe82be7014cdc80b396b2a0a0feab => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:16.715623 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:16.716855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:16.716855 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:16.738150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:16.739433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:16.756167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:16.801197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1396/2750] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:16.811639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:16.813072 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:16.871729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ba0a2-406b-411a-b002-915d01ce25b9 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:16.875522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:16.876126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:16.876397 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:16.876673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:16.886164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:16.887608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:16.898165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1355/2751] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:17.431443 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:17.432923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:17.449267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35871da-fcc7-49f1-af98-c5c7a63d2de1 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:17.452606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:17.453324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:17.453666 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:17.454003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] PATH_INFO: `/v3/projects/2e972a6da88c4def8c30a434f31fb2cf` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:17.455337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e972a6da88c4def8c30a434f31fb2cf)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:17.457733 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:17.584106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e972a6da88c4def8c30a434f31fb2cf', 'request_id': 'req-92b78b74-f617-49b9-9e31-01a051c31c18'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.590579 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e972a6da88c4def8c30a434f31fb2cf', 'request_id': 'req-92b78b74-f617-49b9-9e31-01a051c31c18'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.635089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e972a6da88c4def8c30a434f31fb2cf', 'request_id': 'req-92b78b74-f617-49b9-9e31-01a051c31c18'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.646790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] The token cache is being invalidate because project 2e972a6da88c4def8c30a434f31fb2cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:17.647099 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.647845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.649356 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1397/2752] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:17 2022] DELETE /identity/v3/projects/2e972a6da88c4def8c30a434f31fb2cf => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:17.653582 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:17.663181 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20349f0-a6c7-4e20-86f1-8e5bda130269 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:17.726153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:17.726918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:17.727111 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:17.727351 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] PATH_INFO: `/v3/users/74ff11f4a2364310a06888ab10d2e43e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:17.728492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74ff11f4a2364310a06888ab10d2e43e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:17.730428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:17.758053 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:17.910738 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74ff11f4a2364310a06888ab10d2e43e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.917342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74ff11f4a2364310a06888ab10d2e43e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.918064 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74ff11f4a2364310a06888ab10d2e43e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.918700 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74ff11f4a2364310a06888ab10d2e43e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.972578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-7894380e-fc9a-4600-b49f-f45790da0560 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74ff11f4a2364310a06888ab10d2e43e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:17.990958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1356/2753] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:17 2022] DELETE /identity/v3/users/74ff11f4a2364310a06888ab10d2e43e => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:18.446186 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:18.447673 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:18.461809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b78b74-f617-49b9-9e31-01a051c31c18 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:18.531976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:18.532608 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:18.532881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:18.533151 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] PATH_INFO: `/v3/projects/f853535755d04bb6830733995dac67ae` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:18.534316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f853535755d04bb6830733995dac67ae)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:18.536377 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:18.640853 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:18.642300 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f853535755d04bb6830733995dac67ae', 'request_id': 'req-928a237f-7b7d-4dc5-9e6d-feec8112e988'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:18.648479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f853535755d04bb6830733995dac67ae', 'request_id': 'req-928a237f-7b7d-4dc5-9e6d-feec8112e988'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:18.716020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f853535755d04bb6830733995dac67ae', 'request_id': 'req-928a237f-7b7d-4dc5-9e6d-feec8112e988'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:18.728116 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] The token cache is being invalidate because project f853535755d04bb6830733995dac67ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:18.728406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:18.729067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-928a237f-7b7d-4dc5-9e6d-feec8112e988 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:18.730659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1398/2754] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:18 2022] DELETE /identity/v3/projects/f853535755d04bb6830733995dac67ae => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:18.736248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25f564-61a6-4c68-8dae-94a2df9aa887 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:18.736783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25f564-61a6-4c68-8dae-94a2df9aa887 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:18.736894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25f564-61a6-4c68-8dae-94a2df9aa887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:18.756896 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-eb25f564-61a6-4c68-8dae-94a2df9aa887 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:18.758091 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-eb25f564-61a6-4c68-8dae-94a2df9aa887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:18.816932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1357/2755] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:18.824286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fabd856-11e5-4fcd-a1ab-bd8a5b218aca None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:18.824576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fabd856-11e5-4fcd-a1ab-bd8a5b218aca None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:18.824850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fabd856-11e5-4fcd-a1ab-bd8a5b218aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:18.845449 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-3fabd856-11e5-4fcd-a1ab-bd8a5b218aca None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:18.846692 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3fabd856-11e5-4fcd-a1ab-bd8a5b218aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:18.854501 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1399/2756] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:18.859580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb25f564-61a6-4c68-8dae-94a2df9aa887 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:18.869296 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb25f564-61a6-4c68-8dae-94a2df9aa887 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:18.875311 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:18.875913 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:18.876194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:18.876473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:18.877118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:18.877422 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:18.879112 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:18.891782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1358/2757] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:18.895984 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fabd856-11e5-4fcd-a1ab-bd8a5b218aca None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:18.905233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fabd856-11e5-4fcd-a1ab-bd8a5b218aca None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:18.910037 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:18.910676 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:18.910957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:18.911228 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:18.911859 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:18.913739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:19.014391 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1400/2758] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:18 2022] POST /identity/v3/projects => generated 373 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:19.019142 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:19.030997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c41e56a-abc5-4b1d-9c37-98478ce62623 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:19.088871 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:19.089474 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:19.089750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:19.090019 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:19.090672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:19.092682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:19.194189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1359/2759] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:19 2022] POST /identity/v3/users => generated 287 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:19.198998 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:19.209089 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff86d077-b3e9-4633-8621-b9b31bf26373 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:19.214020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:19.214655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:19.214934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:19.215205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:19.216400 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:19.218196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:19.227826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1401/2760] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:19.236621 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:19.251014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ec8e09-fb3f-4d3b-9b0b-a7b310771aa7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:19.261684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32d60ef-3c57-4576-acc6-851694bc8390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:19.270209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d60ef-3c57-4576-acc6-851694bc8390 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:19.270209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d60ef-3c57-4576-acc6-851694bc8390 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:19.270209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d60ef-3c57-4576-acc6-851694bc8390 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:19.270209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32d60ef-3c57-4576-acc6-851694bc8390 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:19.270209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32d60ef-3c57-4576-acc6-851694bc8390 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:19.278844 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1360/2761] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:19 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:19.283097 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:19.292987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c818be5-c852-498b-9f18-9a7e4c7b1638 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:19.298456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:19.299114 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:19.299393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:19.299667 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] PATH_INFO: `/v3/projects/270b737efc584c729239088e34763f91/users/a941f6c3d05445c58876f16b913259de/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:19.314211 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=270b737efc584c729239088e34763f91, user_id=a941f6c3d05445c58876f16b913259de, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:19.316884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:19.442032 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1402/2762] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:19 2022] PUT /identity/v3/projects/270b737efc584c729239088e34763f91/users/a941f6c3d05445c58876f16b913259de/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:19.453594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06fee8b-ee68-411e-8021-2c18ecb45a00 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:19.454152 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06fee8b-ee68-411e-8021-2c18ecb45a00 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:19.454510 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06fee8b-ee68-411e-8021-2c18ecb45a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:19.469020 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:19.484287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c06fee8b-ee68-411e-8021-2c18ecb45a00 None None] MFA Rules not processed for user `a941f6c3d05445c58876f16b913259de`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:19.485530 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c06fee8b-ee68-411e-8021-2c18ecb45a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:19.564318 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1361/2763] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:19.576828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:19.578738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:19.635461 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d5b3e-8200-43c9-8de7-b76654ff0ec9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:19.640346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58743193-f347-437b-ab1b-f0a68c1e8dca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:19.640949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58743193-f347-437b-ab1b-f0a68c1e8dca service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:19.641227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58743193-f347-437b-ab1b-f0a68c1e8dca service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:19.641499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58743193-f347-437b-ab1b-f0a68c1e8dca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:19.650008 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58743193-f347-437b-ab1b-f0a68c1e8dca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:19.651570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58743193-f347-437b-ab1b-f0a68c1e8dca service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:19.661105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1403/2764] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:19.682890 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06fee8b-ee68-411e-8021-2c18ecb45a00 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:19.744767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06fee8b-ee68-411e-8021-2c18ecb45a00 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:19.749719 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:19.750365 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:19.750672 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:19.750938 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] PATH_INFO: `/v3/users/a941f6c3d05445c58876f16b913259de` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:19.752043 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a941f6c3d05445c58876f16b913259de)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:19.753969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:19.917378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a941f6c3d05445c58876f16b913259de'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:19.925418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a941f6c3d05445c58876f16b913259de'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:19.926087 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a941f6c3d05445c58876f16b913259de'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:19.926722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a941f6c3d05445c58876f16b913259de'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:19.983753 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a941f6c3d05445c58876f16b913259de'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:20.002236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1362/2765] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:19 2022] DELETE /identity/v3/users/a941f6c3d05445c58876f16b913259de => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:20.006771 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:20.007055 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:20.007336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:20.025858 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:20.027093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:20.088161 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1404/2766] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:20.098482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:20.099943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:20.118840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:20.159088 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9173de5-b0b3-4ae9-9fbd-25affe60ed69 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:20.162713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590cb446-92c8-469d-8d63-ad40e1d3fa6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:20.163322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590cb446-92c8-469d-8d63-ad40e1d3fa6f service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:20.163597 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590cb446-92c8-469d-8d63-ad40e1d3fa6f service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:20.163877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590cb446-92c8-469d-8d63-ad40e1d3fa6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:20.173649 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590cb446-92c8-469d-8d63-ad40e1d3fa6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:20.175114 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590cb446-92c8-469d-8d63-ad40e1d3fa6f service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:20.185195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1363/2767] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:20.665002 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:20.666762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:20.679367 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd859fa-46fa-402e-b554-39fb56699ab0 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:20.688575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:20.688575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:20.688575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:20.688575 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] PATH_INFO: `/v3/projects/270b737efc584c729239088e34763f91` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:20.689239 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] RBAC: Authorizing `identity:delete_project(project_id=270b737efc584c729239088e34763f91)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:20.690566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:20.701869 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:20.819326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '270b737efc584c729239088e34763f91', 'request_id': 'req-62dc27f1-2780-40e8-bbfc-68b06edb4a66'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:20.827130 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '270b737efc584c729239088e34763f91', 'request_id': 'req-62dc27f1-2780-40e8-bbfc-68b06edb4a66'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:20.881006 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '270b737efc584c729239088e34763f91', 'request_id': 'req-62dc27f1-2780-40e8-bbfc-68b06edb4a66'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:20.892790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] The token cache is being invalidate because project 270b737efc584c729239088e34763f91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:20.893076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:20.893698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-62dc27f1-2780-40e8-bbfc-68b06edb4a66 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:20.895199 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1405/2768] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:20 2022] DELETE /identity/v3/projects/270b737efc584c729239088e34763f91 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:20.900026 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379faecf-02a4-44bc-be68-38fa510be249 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:20.900501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379faecf-02a4-44bc-be68-38fa510be249 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:20.900615 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379faecf-02a4-44bc-be68-38fa510be249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:20.920883 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-379faecf-02a4-44bc-be68-38fa510be249 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:20.922110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-379faecf-02a4-44bc-be68-38fa510be249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:20.980452 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1364/2769] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:20.988273 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113a43b-1382-4a0a-a919-43f4d3063238 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:20.988557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113a43b-1382-4a0a-a919-43f4d3063238 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:20.988828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113a43b-1382-4a0a-a919-43f4d3063238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.008683 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e113a43b-1382-4a0a-a919-43f4d3063238 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:21.009877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e113a43b-1382-4a0a-a919-43f4d3063238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:21.018729 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1406/2770] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:21.022904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379faecf-02a4-44bc-be68-38fa510be249 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:21.032790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379faecf-02a4-44bc-be68-38fa510be249 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:21.037348 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:21.037943 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.038236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.038535 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.039183 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:21.039484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:21.041062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:21.056809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1365/2771] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 09:01:21.061872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e113a43b-1382-4a0a-a919-43f4d3063238 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:21.071177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e113a43b-1382-4a0a-a919-43f4d3063238 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:21.076312 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:21.076944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.077219 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.077484 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.078096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:21.080007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:21.162390 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1407/2772] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:21 2022] POST /identity/v3/projects => generated 375 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:21.166810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:21.177526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5106577a-0354-428b-b732-0b9f3bb090f8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:21.233795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:21.234395 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.234699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.234970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.235603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:21.237501 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:21.336419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1366/2773] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:21 2022] POST /identity/v3/users => generated 288 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:21.338017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:21.348344 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13248e99-9f1b-4bb6-a5ca-d0cb6f783263 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:21.353378 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:21.359169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.359169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.359169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.359169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:21.359169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:21.367808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1408/2774] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:21.378436 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:21.407944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab85e17-938c-4173-b2b4-565733a03551 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:21.412419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acec6a19-96ab-45df-8578-bd846fd1432a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:21.413038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec6a19-96ab-45df-8578-bd846fd1432a admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.413312 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec6a19-96ab-45df-8578-bd846fd1432a admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.413586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec6a19-96ab-45df-8578-bd846fd1432a admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.414233 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acec6a19-96ab-45df-8578-bd846fd1432a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:21.415859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acec6a19-96ab-45df-8578-bd846fd1432a admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:21.427902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1367/2775] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:21.432092 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:21.442740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bded1ee-c08e-49ef-a911-42ac54c97769 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:21.447323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:21.447968 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.448244 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.448515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] PATH_INFO: `/v3/projects/d475ee4d9d9c40b9b36196cd7563191b/users/df40d8ea5fcf4a9bba048b516e318fa3/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.462637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d475ee4d9d9c40b9b36196cd7563191b, user_id=df40d8ea5fcf4a9bba048b516e318fa3, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:21.465270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:21.553447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1409/2776] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:21 2022] PUT /identity/v3/projects/d475ee4d9d9c40b9b36196cd7563191b/users/df40d8ea5fcf4a9bba048b516e318fa3/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:21.565023 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.565306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.565578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.599096 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] MFA Rules not processed for user `df40d8ea5fcf4a9bba048b516e318fa3`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:21.600236 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:21.613547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:21.666470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1368/2777] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:21.677693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:21.679214 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:21.735897 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a11ff1-4194-408f-8065-7489f3ba62e9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:21.739467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:21.740065 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:21.740340 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:21.740606 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:21.748744 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:21.750145 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:21.758718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1410/2778] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:21 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:27.877334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:27.879079 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:27.987749 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9a753d-c362-4fdd-8399-8c3f638ee1f5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:27.991869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:27.992521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:27.992754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:27.993018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:27.994853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:28.004515 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:28.005957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:28.013892 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1369/2779] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:28.091282 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:28.101971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc649bc5-abc4-4653-a066-8bde84844a0c service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:28.112597 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf2b65-6f69-4ce9-aeec-517886e2fa7e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:28.113210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf2b65-6f69-4ce9-aeec-517886e2fa7e service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:28.113492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf2b65-6f69-4ce9-aeec-517886e2fa7e service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:28.113765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf2b65-6f69-4ce9-aeec-517886e2fa7e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:28.115872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c1cf2b65-6f69-4ce9-aeec-517886e2fa7e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:28.156735 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:28.194287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf2b65-6f69-4ce9-aeec-517886e2fa7e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:28.195816 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf2b65-6f69-4ce9-aeec-517886e2fa7e service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:28.207696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1411/2780] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:28.529310 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:28.530692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:28.596680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e75dfb-fc5f-4382-98bc-05ebb6752c22 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:28.601201 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:28.601825 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:28.602103 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:28.602368 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] PATH_INFO: `/v3/users/152a31d816a948279d974feec2a79e52` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:28.603498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] RBAC: Authorizing `identity:delete_user(user_id=152a31d816a948279d974feec2a79e52)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:28.608108 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:28.818739 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '152a31d816a948279d974feec2a79e52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:28.825427 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '152a31d816a948279d974feec2a79e52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:28.826100 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '152a31d816a948279d974feec2a79e52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:28.826705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '152a31d816a948279d974feec2a79e52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:28.829251 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:28.913099 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '152a31d816a948279d974feec2a79e52'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:28.931234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1370/2781] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:28 2022] DELETE /identity/v3/users/152a31d816a948279d974feec2a79e52 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:28.935879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:28.936241 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:28.936519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:28.969741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:28.969741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:28.971790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:28.973449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:29.039953 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1412/2782] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:29.051710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8c9b56-15ea-49fe-bf63-2f1a2163e750 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:29.055111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:29.056861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:29.057560 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:29.058218 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:29.058668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:29.058944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] PATH_INFO: `/v3/users/df40d8ea5fcf4a9bba048b516e318fa3` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:29.060224 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=df40d8ea5fcf4a9bba048b516e318fa3)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:29.062260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:29.152824 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d73b907-a17a-4282-b9fc-6c20e1c30134 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:29.158053 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182befde-05bd-4967-93ab-0ec0b4dc3b11 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:29.159129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182befde-05bd-4967-93ab-0ec0b4dc3b11 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:29.159509 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182befde-05bd-4967-93ab-0ec0b4dc3b11 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:29.161612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182befde-05bd-4967-93ab-0ec0b4dc3b11 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:29.175669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182befde-05bd-4967-93ab-0ec0b4dc3b11 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:29.178393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182befde-05bd-4967-93ab-0ec0b4dc3b11 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:29.190588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1413/2783] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:29.284726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df40d8ea5fcf4a9bba048b516e318fa3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.292031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df40d8ea5fcf4a9bba048b516e318fa3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.292698 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df40d8ea5fcf4a9bba048b516e318fa3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.293308 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df40d8ea5fcf4a9bba048b516e318fa3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.451013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df40d8ea5fcf4a9bba048b516e318fa3'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.470899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1371/2784] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:28 2022] DELETE /identity/v3/users/df40d8ea5fcf4a9bba048b516e318fa3 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:29.475513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:29.475878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:29.476224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:29.499167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:29.500559 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:29.565906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1414/2785] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:29.575562 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:29.577153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:29.638748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab988c8-9b3f-4e3a-a2c3-441a617398bc admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:29.643381 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:29.643980 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:29.644260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:29.644527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:29.653405 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:29.654907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:29.658807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:29.664941 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1372/2786] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:29.718165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:29.720216 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:29.734589 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3cd86f-8f8a-4a26-bfcc-e6839a9b21e4 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:29.740648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:29.741307 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:29.741596 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:29.741861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] PATH_INFO: `/v3/projects/b7ba426889f548129d1c9ad246051754` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:29.743670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7ba426889f548129d1c9ad246051754)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:29.746140 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:29.748082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:29.906090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7ba426889f548129d1c9ad246051754', 'request_id': 'req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.913679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7ba426889f548129d1c9ad246051754', 'request_id': 'req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.981547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7ba426889f548129d1c9ad246051754', 'request_id': 'req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.993015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] The token cache is being invalidate because project b7ba426889f548129d1c9ad246051754 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:29.993382 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.994096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:29.995752 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1415/2787] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:29 2022] DELETE /identity/v3/projects/b7ba426889f548129d1c9ad246051754 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:29.999748 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:30.010655 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb262e2-fd1b-41ee-8797-5fd7db795bbf service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:30.066531 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:30.068172 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:30.071618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:30.072258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.072539 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.072814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] PATH_INFO: `/v3/users/0d8d27c70f9d4697bbde2eb1cd2b5344` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.073947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d8d27c70f9d4697bbde2eb1cd2b5344)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:30.075932 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:30.081593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33bbad-2d9d-4b60-a5ce-7610e35256c5 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:30.089306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:30.090038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.095958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.095958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] PATH_INFO: `/v3/projects/d475ee4d9d9c40b9b36196cd7563191b` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.097435 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d475ee4d9d9c40b9b36196cd7563191b)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:30.099468 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:30.317765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd475ee4d9d9c40b9b36196cd7563191b', 'request_id': 'req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.321718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d8d27c70f9d4697bbde2eb1cd2b5344'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.325878 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd475ee4d9d9c40b9b36196cd7563191b', 'request_id': 'req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.330242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d8d27c70f9d4697bbde2eb1cd2b5344'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.331117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d8d27c70f9d4697bbde2eb1cd2b5344'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.331722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d8d27c70f9d4697bbde2eb1cd2b5344'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.441912 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd475ee4d9d9c40b9b36196cd7563191b', 'request_id': 'req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.453236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] The token cache is being invalidate because project d475ee4d9d9c40b9b36196cd7563191b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:30.453577 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.455379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-69fd15be-16f7-48f3-9c87-ac06afeb4bf7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.457007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1416/2788] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:30 2022] DELETE /identity/v3/projects/d475ee4d9d9c40b9b36196cd7563191b => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:30.461710 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e44c270-1a0c-4b46-a991-ed2794a266cc None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.461987 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e44c270-1a0c-4b46-a991-ed2794a266cc None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.462267 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e44c270-1a0c-4b46-a991-ed2794a266cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.483321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-8e44c270-1a0c-4b46-a991-ed2794a266cc None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:30.484562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8e44c270-1a0c-4b46-a991-ed2794a266cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:30.521645 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e6cf690e-65ac-42a3-8cb3-d098bede3f46 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d8d27c70f9d4697bbde2eb1cd2b5344'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:30.548969 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1373/2789] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:29 2022] DELETE /identity/v3/users/0d8d27c70f9d4697bbde2eb1cd2b5344 => generated 0 bytes in 547 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:30.549341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1417/2790] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:30.556906 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9896320-fa0d-4705-aaff-d1678169705d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.557188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9896320-fa0d-4705-aaff-d1678169705d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.557457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9896320-fa0d-4705-aaff-d1678169705d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.580937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-a9896320-fa0d-4705-aaff-d1678169705d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:30.582189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-a9896320-fa0d-4705-aaff-d1678169705d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:30.666342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1374/2791] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:30.669288 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e44c270-1a0c-4b46-a991-ed2794a266cc None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:30.679658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e44c270-1a0c-4b46-a991-ed2794a266cc None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:30.687801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:30.689153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.689319 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.689590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.690347 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:30.690717 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:30.693373 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:30.701328 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:30.705480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1418/2792] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 09:01:30.709958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9896320-fa0d-4705-aaff-d1678169705d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:30.719249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9896320-fa0d-4705-aaff-d1678169705d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:30.723930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:30.724519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.724795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.725063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.725695 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:30.727590 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:30.744958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:30.838300 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1375/2793] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:30 2022] POST /identity/v3/projects => generated 365 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:30.841694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:30.853271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d54d0d-e77b-4bbc-9edd-fe50d14a98fc admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:30.913014 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:30.913662 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.914790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.915631 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.916360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:30.918274 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:30.968458 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:30.970973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:30.987665 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399e1cde-b2f6-4930-a222-1b1b93e3323e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:30.995817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:30.996102 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1419/2794] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:30 2022] POST /identity/v3/users => generated 283 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:30.996673 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:30.996949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:30.997221 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] PATH_INFO: `/v3/projects/543deab5257944d8ad67b5fe5cf3fbf0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:30.998379 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] RBAC: Authorizing `identity:delete_project(project_id=543deab5257944d8ad67b5fe5cf3fbf0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.001341 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.006649 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.019895 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd71975-d8c0-4f76-86ff-92337fa0939a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.025245 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.025780 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.026164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.026580 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.027471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.029246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.042000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1420/2795] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:31 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:31.046000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.056817 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abb43b7-4c3a-4d60-87c6-be18a3518780 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.061700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.062286 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.062681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.063640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.064388 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.066007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.076815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1421/2796] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:31.081352 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.085626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '543deab5257944d8ad67b5fe5cf3fbf0', 'request_id': 'req-e487e795-a127-4103-90d9-9e960a0c5944'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.091457 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e132c0c3-f9eb-4909-bb05-44355d8c26a1 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.151931 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341919c5-9f88-4bdb-8a23-55ce6319cc7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.152605 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341919c5-9f88-4bdb-8a23-55ce6319cc7a admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.152880 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341919c5-9f88-4bdb-8a23-55ce6319cc7a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.153156 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341919c5-9f88-4bdb-8a23-55ce6319cc7a admin admin] PATH_INFO: `/v3/projects/3bb12f391098493b9bc69d7e7e5864d4/users/7aeb97b19f12450fb3634eea8d5f1714/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.153313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '543deab5257944d8ad67b5fe5cf3fbf0', 'request_id': 'req-e487e795-a127-4103-90d9-9e960a0c5944'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.161918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '543deab5257944d8ad67b5fe5cf3fbf0', 'request_id': 'req-e487e795-a127-4103-90d9-9e960a0c5944'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.176163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341919c5-9f88-4bdb-8a23-55ce6319cc7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bb12f391098493b9bc69d7e7e5864d4, user_id=7aeb97b19f12450fb3634eea8d5f1714, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.176623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] The token cache is being invalidate because project 543deab5257944d8ad67b5fe5cf3fbf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:31.177086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.177974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-e487e795-a127-4103-90d9-9e960a0c5944 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.179035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341919c5-9f88-4bdb-8a23-55ce6319cc7a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.179854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1376/2797] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:30 2022] DELETE /identity/v3/projects/543deab5257944d8ad67b5fe5cf3fbf0 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:31.184632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80340f-39d4-436e-97d4-77c5e7ff760c None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.184931 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80340f-39d4-436e-97d4-77c5e7ff760c None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.185398 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80340f-39d4-436e-97d4-77c5e7ff760c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.207126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-6b80340f-39d4-436e-97d4-77c5e7ff760c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:31.208261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6b80340f-39d4-436e-97d4-77c5e7ff760c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:31.217407 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1377/2798] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:31.227248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f8c422-fa69-4f24-b79c-2bb5eb587b35 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.227708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f8c422-fa69-4f24-b79c-2bb5eb587b35 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.227967 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f8c422-fa69-4f24-b79c-2bb5eb587b35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.250929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-82f8c422-fa69-4f24-b79c-2bb5eb587b35 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:31.253677 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-82f8c422-fa69-4f24-b79c-2bb5eb587b35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:31.266974 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1378/2799] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:31.268916 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f8c422-fa69-4f24-b79c-2bb5eb587b35 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.285442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1422/2800] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:31 2022] PUT /identity/v3/projects/3bb12f391098493b9bc69d7e7e5864d4/users/7aeb97b19f12450fb3634eea8d5f1714/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:31.285594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f8c422-fa69-4f24-b79c-2bb5eb587b35 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.286876 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.287189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.287452 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.331304 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] MFA Rules not processed for user `7aeb97b19f12450fb3634eea8d5f1714`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:31.333405 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:31.392137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:31.400256 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.400907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.401186 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.401716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.402109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:31.402445 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.407710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.421571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1379/2801] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:31.427657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.430791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1423/2802] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:31 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:31.440684 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b8c9b2-59c6-4e66-833b-ac37f4a63b70 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.446702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.446933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.447527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.447904 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.448275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.448503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:31.449816 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.452162 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.522144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0fc65-bfab-4443-89a1-571289a698d3 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.527163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.527944 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.528067 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.528336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.538285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.540064 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.562721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1424/2803] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:31 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:31.566371 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1380/2804] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:31 2022] POST /identity/v3/projects => generated 393 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:31.567536 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.588412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413c0303-ca5f-43a9-823a-d83b45d76095 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.657691 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.658421 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.658701 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.659070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.659583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.661538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.722860 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.724685 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:31.737854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aadc175-5497-4cb9-868a-5ec8104334fa admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.742511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.746354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.746354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.746354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] PATH_INFO: `/v3/users/fd98ae518f0d4956a0c08ca8caaeebac` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.746354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd98ae518f0d4956a0c08ca8caaeebac)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.746991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.801476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1425/2805] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:31 2022] POST /identity/v3/users => generated 297 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:31.805739 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.815689 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604745e8-2420-4579-9a28-1ee5b8920627 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.820480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.821075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.821351 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.821617 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.822290 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.824614 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.838948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1426/2806] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:01:31.843896 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.850927 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43f4427-a190-417b-a610-c30a828c7b54 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.855513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.856080 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.856353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.856621 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.857246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.858850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.869129 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1427/2807] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:01:31.873210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:31.883158 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b784c23-e560-43c4-b407-74bc205574bb admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:31.887711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b1a75-e882-4727-99f9-0d675eab6768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:31.888317 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b1a75-e882-4727-99f9-0d675eab6768 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.888619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b1a75-e882-4727-99f9-0d675eab6768 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.888885 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b1a75-e882-4727-99f9-0d675eab6768 admin admin] PATH_INFO: `/v3/projects/f5980440f8b04892affd1f007cb5ce64/users/3df696cb8da0448d9077b6de900fb44a/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.904103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286b1a75-e882-4727-99f9-0d675eab6768 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5980440f8b04892affd1f007cb5ce64, user_id=3df696cb8da0448d9077b6de900fb44a, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:31.907170 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286b1a75-e882-4727-99f9-0d675eab6768 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:31.972578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:31.976609 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1428/2808] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:31 2022] PUT /identity/v3/projects/f5980440f8b04892affd1f007cb5ce64/users/3df696cb8da0448d9077b6de900fb44a/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:31.984327 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd98ae518f0d4956a0c08ca8caaeebac'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.986962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:31.987262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:31.987517 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:31.991784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd98ae518f0d4956a0c08ca8caaeebac'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.992354 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd98ae518f0d4956a0c08ca8caaeebac'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:31.992950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd98ae518f0d4956a0c08ca8caaeebac'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:32.016857 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] MFA Rules not processed for user `3df696cb8da0448d9077b6de900fb44a`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:32.018078 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:32.030955 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:32.058061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-34881972-3fed-48b9-975c-ce8a3e761577 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd98ae518f0d4956a0c08ca8caaeebac'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:32.077276 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1381/2809] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:31 2022] DELETE /identity/v3/users/fd98ae518f0d4956a0c08ca8caaeebac => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:32.082275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.082581 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.082854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.087127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1429/2810] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:31 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:32.091259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.092554 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:32.102118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:32.103333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:32.159372 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79af1bed-739b-4245-ac2c-3f2205f886a3 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.163250 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1382/2811] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:32.164652 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.165316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.165601 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.165872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.166639 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.168733 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.173234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.174614 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:32.252018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13e6d13-b989-4b22-a279-03f6a75616b1 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.266732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.266732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.266732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.266732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.268459 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.270049 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.279949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1383/2812] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:32.291699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1430/2813] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:32 2022] POST /identity/v3/projects => generated 391 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:32.301876 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.312322 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44433fb-ff3f-4d4e-93b8-2eb662a29ac6 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.398875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.399497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.400970 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.401317 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.401998 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.403954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.521689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:32.527867 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1384/2814] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:32 2022] POST /identity/v3/users => generated 302 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:32.532305 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.543492 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33be3cf2-913c-46ae-b21d-940cc3e5820b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.549769 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.550385 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.550702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.557176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.557176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.557176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.566760 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1431/2815] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:32.570518 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.586032 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d3e49c-eeae-4f3f-adb0-50e8de5f53a9 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.586853 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.587636 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.587982 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.588328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] PATH_INFO: `/v3/projects/2210622b6120492bb617941d963669d2/users/3721151df6d94f35b4189fcbfe081674/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.606369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2210622b6120492bb617941d963669d2, user_id=3721151df6d94f35b4189fcbfe081674, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.609131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.728283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1385/2816] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:32 2022] PUT /identity/v3/projects/2210622b6120492bb617941d963669d2/users/3721151df6d94f35b4189fcbfe081674/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:32.732306 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.743520 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56375b4e-d3b8-46a8-af2e-16b493ab21f8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.808197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.808798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.809074 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.809343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.809979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.811587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.821686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1432/2817] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:32.825902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.831494 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.833339 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:32.836806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e4492a-0a96-4a81-84a7-c89cf43c0e13 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.843732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dcbefb-9416-406e-9383-add3fecad1e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.847826 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117940f3-cdde-4dbc-99fe-e19db3b2d764 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:32.852970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:32.853714 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.854222 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.854576 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] PATH_INFO: `/v3/projects/b5f9accd3435416d9749ccab9b4e4d9d` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.854907 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcbefb-9416-406e-9383-add3fecad1e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.855192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcbefb-9416-406e-9383-add3fecad1e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.855468 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dcbefb-9416-406e-9383-add3fecad1e8 admin admin] PATH_INFO: `/v3/projects/2210622b6120492bb617941d963669d2/users/3721151df6d94f35b4189fcbfe081674/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.855986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5f9accd3435416d9749ccab9b4e4d9d)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.858118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dcbefb-9416-406e-9383-add3fecad1e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2210622b6120492bb617941d963669d2, user_id=3721151df6d94f35b4189fcbfe081674, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:32.858462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.860930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dcbefb-9416-406e-9383-add3fecad1e8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:32.872504 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1386/2818] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:32 2022] PUT /identity/v3/projects/2210622b6120492bb617941d963669d2/users/3721151df6d94f35b4189fcbfe081674/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:32.884676 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:32.884958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:32.885234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:32.885923 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:32.913619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] MFA Rules not processed for user `3721151df6d94f35b4189fcbfe081674`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:32.914874 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:32.974465 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5f9accd3435416d9749ccab9b4e4d9d', 'request_id': 'req-e40dad29-7ca0-4e68-a07c-883b3829052a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:32.981248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5f9accd3435416d9749ccab9b4e4d9d', 'request_id': 'req-e40dad29-7ca0-4e68-a07c-883b3829052a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:32.982920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1387/2819] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:32 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:32.994524 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:32.995921 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:33.038660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5f9accd3435416d9749ccab9b4e4d9d', 'request_id': 'req-e40dad29-7ca0-4e68-a07c-883b3829052a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:33.050637 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] The token cache is being invalidate because project b5f9accd3435416d9749ccab9b4e4d9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:33.050938 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:33.051607 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:33.053318 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1433/2820] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:32 2022] DELETE /identity/v3/projects/b5f9accd3435416d9749ccab9b4e4d9d => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:33.054785 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc797a30-06b2-422e-8de6-76e586c29cd2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:33.116013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:33.122285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1f4e06-28e1-4aae-88df-7a2db7e80959 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:33.126791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f4e06-28e1-4aae-88df-7a2db7e80959 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:33.126791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f4e06-28e1-4aae-88df-7a2db7e80959 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:33.126791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f4e06-28e1-4aae-88df-7a2db7e80959 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:33.126791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5f1f4e06-28e1-4aae-88df-7a2db7e80959 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:33.198174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1f4e06-28e1-4aae-88df-7a2db7e80959 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:33.199679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1f4e06-28e1-4aae-88df-7a2db7e80959 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:33.210746 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1388/2821] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:32 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:01:33.260670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:33.262227 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:33.276122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40dad29-7ca0-4e68-a07c-883b3829052a admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:33.280479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:33.281392 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:33.281761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:33.282165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:33.284345 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:33.349059 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:33.350521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:33.359325 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1434/2822] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:33 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:33.646084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573e97d-8bb2-451d-9b63-fef1c4f3a44b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:33.646473 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573e97d-8bb2-451d-9b63-fef1c4f3a44b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:33.646793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573e97d-8bb2-451d-9b63-fef1c4f3a44b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:33.666588 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-c573e97d-8bb2-451d-9b63-fef1c4f3a44b None None] MFA Rules not processed for user `471299b556bd4c22b51defc14b49d8eb`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:33.667799 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-c573e97d-8bb2-451d-9b63-fef1c4f3a44b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:33.727603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1389/2823] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:38.502556 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:38.504088 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:38.582124 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1edeac-0ebf-408c-8f7a-cf3663fd8b95 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:38.589431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:38.590223 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:38.590604 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:38.590943 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] PATH_INFO: `/v3/users/3df696cb8da0448d9077b6de900fb44a` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:38.592235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3df696cb8da0448d9077b6de900fb44a)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:38.594316 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:38.801811 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3df696cb8da0448d9077b6de900fb44a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:38.807972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:38.818502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3df696cb8da0448d9077b6de900fb44a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:38.818502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3df696cb8da0448d9077b6de900fb44a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:38.818502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3df696cb8da0448d9077b6de900fb44a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:38.943945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3df696cb8da0448d9077b6de900fb44a'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:38.965000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1435/2824] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:38 2022] DELETE /identity/v3/users/3df696cb8da0448d9077b6de900fb44a => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:38.969803 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:38.970832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:38.971292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:38.992451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:38.993663 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:39.059316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1390/2825] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:39.069573 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:39.070942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:39.133038 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a107e31e-d42f-4cf0-bebe-15ee2db52140 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:39.136659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:39.137266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:39.137538 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:39.137803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:39.147850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:39.149271 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:39.159399 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1436/2826] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:39.664788 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:39.666271 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:39.683777 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da70c152-877f-4bc4-a33a-6160d4061f3d None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:39.688242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:39.690151 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:39.690453 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:39.690734 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] PATH_INFO: `/v3/projects/f5980440f8b04892affd1f007cb5ce64` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:39.693230 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5980440f8b04892affd1f007cb5ce64)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:39.695522 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:39.823702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:39.840051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5980440f8b04892affd1f007cb5ce64', 'request_id': 'req-d5cc48c9-cbbe-4092-82b6-be06de06864d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:39.900517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5980440f8b04892affd1f007cb5ce64', 'request_id': 'req-d5cc48c9-cbbe-4092-82b6-be06de06864d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:39.911959 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5980440f8b04892affd1f007cb5ce64', 'request_id': 'req-d5cc48c9-cbbe-4092-82b6-be06de06864d'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:39.930790 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] The token cache is being invalidate because project f5980440f8b04892affd1f007cb5ce64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:39.933190 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:39.934784 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:39.938262 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1391/2827] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:39 2022] DELETE /identity/v3/projects/f5980440f8b04892affd1f007cb5ce64 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:39.943863 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:39.957990 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88887bb4-0e93-41cf-81b5-ea71dc5bfc86 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:40.044792 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:40.045423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:40.045696 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:40.045966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] PATH_INFO: `/v3/users/3721151df6d94f35b4189fcbfe081674` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:40.047184 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] RBAC: Authorizing `identity:delete_user(user_id=3721151df6d94f35b4189fcbfe081674)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:40.049785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:40.265632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3721151df6d94f35b4189fcbfe081674'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:40.273807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3721151df6d94f35b4189fcbfe081674'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:40.274693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3721151df6d94f35b4189fcbfe081674'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:40.275326 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3721151df6d94f35b4189fcbfe081674'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:40.398900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-debbd34a-3cda-4f6e-8d80-d5042f6ccdcd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3721151df6d94f35b4189fcbfe081674'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:40.409513 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:40.421588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1437/2828] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:39 2022] DELETE /identity/v3/users/3721151df6d94f35b4189fcbfe081674 => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:40.940619 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:40.944083 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:40.957608 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cc48c9-cbbe-4092-82b6-be06de06864d admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:41.016192 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:41.016899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.017733 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.018037 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] PATH_INFO: `/v3/projects/2210622b6120492bb617941d963669d2` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.019234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=2210622b6120492bb617941d963669d2)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:41.023118 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:41.149518 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2210622b6120492bb617941d963669d2', 'request_id': 'req-a78cf621-3494-42d9-a372-8124ab8938ce'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:41.240248 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2210622b6120492bb617941d963669d2', 'request_id': 'req-a78cf621-3494-42d9-a372-8124ab8938ce'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:41.250858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2210622b6120492bb617941d963669d2', 'request_id': 'req-a78cf621-3494-42d9-a372-8124ab8938ce'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:41.263797 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] The token cache is being invalidate because project 2210622b6120492bb617941d963669d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:41.265385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:41.265385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-a78cf621-3494-42d9-a372-8124ab8938ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:41.267466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1392/2829] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:40 2022] DELETE /identity/v3/projects/2210622b6120492bb617941d963669d2 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:41.272438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9207e7-4650-4ed7-8e24-9f901b9341fd None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.272898 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9207e7-4650-4ed7-8e24-9f901b9341fd None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.272996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9207e7-4650-4ed7-8e24-9f901b9341fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.293961 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-df9207e7-4650-4ed7-8e24-9f901b9341fd None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:41.295323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-df9207e7-4650-4ed7-8e24-9f901b9341fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:41.356350 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1438/2830] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 09:01:41.364204 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85529715-d598-4362-970f-95eed9e970b2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.364519 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85529715-d598-4362-970f-95eed9e970b2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.364795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85529715-d598-4362-970f-95eed9e970b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.385039 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-85529715-d598-4362-970f-95eed9e970b2 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:41.386297 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-85529715-d598-4362-970f-95eed9e970b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:41.395174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1393/2831] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:41.399515 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9207e7-4650-4ed7-8e24-9f901b9341fd None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:41.410234 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9207e7-4650-4ed7-8e24-9f901b9341fd None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:41.414818 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:41.415414 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.415698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.415970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.416619 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:41.416920 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:41.418529 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:41.430706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1439/2832] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:41.435120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85529715-d598-4362-970f-95eed9e970b2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:41.444679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85529715-d598-4362-970f-95eed9e970b2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:41.449394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:41.450051 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.450325 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.450639 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.451279 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:41.453168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:41.539854 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1394/2833] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:41 2022] POST /identity/v3/projects => generated 399 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 09:01:41.543900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:41.553741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d1356-4346-4150-8027-ba61e02342d8 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:41.609370 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:41.609966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.610246 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.610581 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.611284 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:41.613217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:41.768125 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1440/2834] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:41 2022] POST /identity/v3/users => generated 300 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:41.772234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:41.783116 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0076c-efb8-45be-9e5d-a06f539863c0 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:41.787732 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:41.788390 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.788689 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.788958 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.789594 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:41.791260 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:41.801521 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1395/2835] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:41.805426 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:41.816471 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85a667c-1dcd-461e-911a-31e7d1c3aae5 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:41.821137 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54671f9-920b-4c29-8d8a-18cef3edfd87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:41.821732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54671f9-920b-4c29-8d8a-18cef3edfd87 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.822010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54671f9-920b-4c29-8d8a-18cef3edfd87 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.822283 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54671f9-920b-4c29-8d8a-18cef3edfd87 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.822954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54671f9-920b-4c29-8d8a-18cef3edfd87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:41.824570 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54671f9-920b-4c29-8d8a-18cef3edfd87 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:41.836081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1441/2836] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:41.839949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:41.849573 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b828be39-a183-4db9-acf7-1c00c95fcc23 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:41.854008 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:41.854671 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.854950 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.855220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] PATH_INFO: `/v3/projects/25f649187dc54a099583dafa8f92e8b5/users/724861b22fc04c1ca9f8e15e04776ef7/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:41.867909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:41.870810 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] RBAC: Authorizing `identity:create_grant(project_id=25f649187dc54a099583dafa8f92e8b5, user_id=724861b22fc04c1ca9f8e15e04776ef7, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:41.873435 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:41.973243 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1396/2837] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:41 2022] PUT /identity/v3/projects/25f649187dc54a099583dafa8f92e8b5/users/724861b22fc04c1ca9f8e15e04776ef7/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:41.983535 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a2c658-a99f-4108-9c30-3473b329b2df None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:41.983814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a2c658-a99f-4108-9c30-3473b329b2df None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:41.984082 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a2c658-a99f-4108-9c30-3473b329b2df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.013193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d3a2c658-a99f-4108-9c30-3473b329b2df None None] MFA Rules not processed for user `724861b22fc04c1ca9f8e15e04776ef7`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:42.014432 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d3a2c658-a99f-4108-9c30-3473b329b2df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:42.059240 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:42.080103 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1442/2838] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:41 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:42.084565 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:42.145633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83970f82-f707-46fd-903c-e438e021e136 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:42.150208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:42.150837 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.151110 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.151374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.151993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:42.153855 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:42.228215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1397/2839] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:42 2022] POST /identity/v3/projects => generated 399 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:42.232412 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a2c658-a99f-4108-9c30-3473b329b2df None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:42.243505 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a2c658-a99f-4108-9c30-3473b329b2df None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:42.303189 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:42.303914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.304017 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.304230 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.304901 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:42.306821 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:42.450237 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1443/2840] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:42 2022] POST /identity/v3/users => generated 306 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:42.458220 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:42.468450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec853f4f-0bd2-4a1f-b040-61b299e8ccd3 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:42.473922 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:42.474616 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.474894 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.475164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.475802 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:42.477386 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:42.488421 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1398/2841] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:42.492665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:42.502679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56293e1a-8c6f-4a6d-95ae-0e0d9ff27ca2 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:42.507197 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:42.507836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.508111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.508379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] PATH_INFO: `/v3/projects/9e9f52aae6504525b1e82382c5898dbe/users/663a23b01b34463cbb715be4c3de20f5/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.522851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e9f52aae6504525b1e82382c5898dbe, user_id=663a23b01b34463cbb715be4c3de20f5, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:42.525460 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:42.593023 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1444/2842] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:42 2022] PUT /identity/v3/projects/9e9f52aae6504525b1e82382c5898dbe/users/663a23b01b34463cbb715be4c3de20f5/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:42.597574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:42.607419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34d73b0-b2fc-4196-9a1b-8321d9fc3c48 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:42.663188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76e497d-9b16-4b1e-b58d-6026359b7791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:42.663895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76e497d-9b16-4b1e-b58d-6026359b7791 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.664177 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76e497d-9b16-4b1e-b58d-6026359b7791 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.664447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76e497d-9b16-4b1e-b58d-6026359b7791 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.665080 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76e497d-9b16-4b1e-b58d-6026359b7791 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:42.666759 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76e497d-9b16-4b1e-b58d-6026359b7791 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:42.678770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1399/2843] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:42.681812 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:42.692289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c60bce-2f8f-4354-9abf-23178e57c830 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:42.697259 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:42.697914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.698193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.698502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] PATH_INFO: `/v3/projects/9e9f52aae6504525b1e82382c5898dbe/users/663a23b01b34463cbb715be4c3de20f5/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.700450 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e9f52aae6504525b1e82382c5898dbe, user_id=663a23b01b34463cbb715be4c3de20f5, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:42.703155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:42.717164 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1445/2844] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:42 2022] PUT /identity/v3/projects/9e9f52aae6504525b1e82382c5898dbe/users/663a23b01b34463cbb715be4c3de20f5/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:42.784508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb8b7b1-86af-462d-bda9-e00bc10e2043 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.784791 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb8b7b1-86af-462d-bda9-e00bc10e2043 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.785086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb8b7b1-86af-462d-bda9-e00bc10e2043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.812991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0cb8b7b1-86af-462d-bda9-e00bc10e2043 None None] MFA Rules not processed for user `663a23b01b34463cbb715be4c3de20f5`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:42.814394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0cb8b7b1-86af-462d-bda9-e00bc10e2043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:42.861143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:42.881927 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1400/2845] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:42 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:42.893918 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:42.895640 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:42.952904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3584b969-7824-485c-b641-d2ff65744f4e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:42.957266 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:42.957847 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:42.958121 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:42.958386 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:42.967127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:42.968535 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:42.978456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1446/2846] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:42 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:43.161013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb8b7b1-86af-462d-bda9-e00bc10e2043 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:43.171403 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb8b7b1-86af-462d-bda9-e00bc10e2043 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:43.175555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f1b9a9-1a02-4cfb-8b4d-3e1340e14258 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:43.182280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f1b9a9-1a02-4cfb-8b4d-3e1340e14258 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:43.182280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f1b9a9-1a02-4cfb-8b4d-3e1340e14258 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:43.182280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f1b9a9-1a02-4cfb-8b4d-3e1340e14258 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:43.238555 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f1b9a9-1a02-4cfb-8b4d-3e1340e14258 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:43.238688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f1b9a9-1a02-4cfb-8b4d-3e1340e14258 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:43.248120 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1401/2847] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:43 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:48.556861 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:48.603042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:48.628664 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c777996-e8ce-4a21-914f-a2cd808b7e5c service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:48.635025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:48.635677 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:48.635960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:48.636287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] PATH_INFO: `/v3/users/724861b22fc04c1ca9f8e15e04776ef7` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:48.637433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=724861b22fc04c1ca9f8e15e04776ef7)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:48.642967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:48.827314 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '724861b22fc04c1ca9f8e15e04776ef7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:48.835431 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '724861b22fc04c1ca9f8e15e04776ef7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:48.836353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '724861b22fc04c1ca9f8e15e04776ef7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:48.836958 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '724861b22fc04c1ca9f8e15e04776ef7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:48.920933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '724861b22fc04c1ca9f8e15e04776ef7'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:48.939091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1447/2848] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:48 2022] DELETE /identity/v3/users/724861b22fc04c1ca9f8e15e04776ef7 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:48.943646 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:48.944242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:48.944242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:48.963795 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:48.965001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:49.031680 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1402/2849] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:49.043324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:49.044807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:49.111940 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d46b23b-469b-4030-a2df-6489df395e7b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:49.115681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:49.116299 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:49.117233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:49.117406 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:49.131167 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:49.133971 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:49.147801 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1448/2850] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:49.604580 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:49.606196 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:49.621585 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca86d803-be53-4841-a1f0-bc3ab0cf8fb3 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:49.627475 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:49.628195 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:49.628482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:49.628756 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] PATH_INFO: `/v3/projects/25f649187dc54a099583dafa8f92e8b5` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:49.629937 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] RBAC: Authorizing `identity:delete_project(project_id=25f649187dc54a099583dafa8f92e8b5)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:49.632005 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:49.743281 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25f649187dc54a099583dafa8f92e8b5', 'request_id': 'req-6531509c-ee06-4818-af0e-59a891abe209'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:49.821450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25f649187dc54a099583dafa8f92e8b5', 'request_id': 'req-6531509c-ee06-4818-af0e-59a891abe209'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:49.832574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25f649187dc54a099583dafa8f92e8b5', 'request_id': 'req-6531509c-ee06-4818-af0e-59a891abe209'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:49.851688 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] The token cache is being invalidate because project 25f649187dc54a099583dafa8f92e8b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:49.854750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:49.854750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:49.854750 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1403/2851] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:49 2022] DELETE /identity/v3/projects/25f649187dc54a099583dafa8f92e8b5 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:49.859025 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:49.869270 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c490a0c-1c72-4b6a-9be2-578502ae49d0 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:49.934754 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:49.935302 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:49.935582 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:49.935841 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] PATH_INFO: `/v3/users/663a23b01b34463cbb715be4c3de20f5` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:49.936960 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=663a23b01b34463cbb715be4c3de20f5)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:49.939422 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:49.942543 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:50.106909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '663a23b01b34463cbb715be4c3de20f5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:50.114360 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '663a23b01b34463cbb715be4c3de20f5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:50.115070 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '663a23b01b34463cbb715be4c3de20f5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:50.115672 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '663a23b01b34463cbb715be4c3de20f5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:50.210546 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-f6532882-5360-45f7-adc6-49e7f9cfaea9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '663a23b01b34463cbb715be4c3de20f5'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:50.229439 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1449/2852] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:49 2022] DELETE /identity/v3/users/663a23b01b34463cbb715be4c3de20f5 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:50.856823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:50.858610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:50.871809 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6531509c-ee06-4818-af0e-59a891abe209 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:50.877205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:50.929062 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:50.929721 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:50.929997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:50.930265 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] PATH_INFO: `/v3/projects/9e9f52aae6504525b1e82382c5898dbe` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:50.931466 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e9f52aae6504525b1e82382c5898dbe)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:50.935859 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:51.061708 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e9f52aae6504525b1e82382c5898dbe', 'request_id': 'req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:51.132712 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e9f52aae6504525b1e82382c5898dbe', 'request_id': 'req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:51.149754 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e9f52aae6504525b1e82382c5898dbe', 'request_id': 'req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:51.165295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] The token cache is being invalidate because project 9e9f52aae6504525b1e82382c5898dbe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:51.165589 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:51.166329 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-9bf26ab9-1231-48e8-9c6c-07e0e0ef89ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:51.168021 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1404/2853] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:50 2022] DELETE /identity/v3/projects/9e9f52aae6504525b1e82382c5898dbe => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:51.174790 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0779388-7209-4caf-bea0-dc0e2506ce1c None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.175768 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0779388-7209-4caf-bea0-dc0e2506ce1c None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.176105 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0779388-7209-4caf-bea0-dc0e2506ce1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.196502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-c0779388-7209-4caf-bea0-dc0e2506ce1c None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:51.197697 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c0779388-7209-4caf-bea0-dc0e2506ce1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:51.263193 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1450/2854] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:51.271836 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf35f5c-89ed-4d10-accc-f986d8b85b93 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.272146 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf35f5c-89ed-4d10-accc-f986d8b85b93 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.272419 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf35f5c-89ed-4d10-accc-f986d8b85b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.293579 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-9bf35f5c-89ed-4d10-accc-f986d8b85b93 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:51.297434 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-9bf35f5c-89ed-4d10-accc-f986d8b85b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:51.303223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1405/2855] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:51.307508 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0779388-7209-4caf-bea0-dc0e2506ce1c None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:51.319010 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0779388-7209-4caf-bea0-dc0e2506ce1c None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:51.323834 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:51.324442 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.324721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.324994 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] PATH_INFO: `/v3/domains` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.325687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88203) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:51.325986 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:51.327657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:51.339877 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1451/2856] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:51.344571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf35f5c-89ed-4d10-accc-f986d8b85b93 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:51.354423 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf35f5c-89ed-4d10-accc-f986d8b85b93 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:51.359369 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:51.360031 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.360314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.360583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.361211 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:51.364197 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:51.457440 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1406/2857] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:51 2022] POST /identity/v3/projects => generated 375 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:51.458660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:51.468810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8757bb5-3b5e-4bb6-be62-ed8509d7021e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:51.552117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:51.552423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.552765 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.553098 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.553798 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:51.555810 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:51.650490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:51.656479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1452/2858] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:51 2022] POST /identity/v3/users => generated 288 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:51.662267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:51.673292 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649a2a0c-af7c-4599-aee8-4b7c584f334b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:51.678184 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:51.678823 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.679095 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.679358 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.679999 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:51.681569 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:51.691954 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1407/2859] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:51.695972 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:51.706979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7b17b-2bfb-4e4c-aa5d-8496d45fdf44 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:51.711942 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655d610c-50ae-4095-b4b1-6ed7f79aad3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:51.714914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d610c-50ae-4095-b4b1-6ed7f79aad3b admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.714914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d610c-50ae-4095-b4b1-6ed7f79aad3b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.714914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d610c-50ae-4095-b4b1-6ed7f79aad3b admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.714914 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655d610c-50ae-4095-b4b1-6ed7f79aad3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:51.715506 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655d610c-50ae-4095-b4b1-6ed7f79aad3b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:51.726491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1453/2860] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:51.731044 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:51.742048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad9bcef-d522-4724-adc7-fc3e2ac25fce admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:51.746725 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:51.747378 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.747651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.747920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] PATH_INFO: `/v3/projects/6cab1ea5ecc34b728f47c95d6fcc1d87/users/ba6483da3d7945cfafca7cae9082f4e7/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.763259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6cab1ea5ecc34b728f47c95d6fcc1d87, user_id=ba6483da3d7945cfafca7cae9082f4e7, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:51.766679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:51.841533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1408/2861] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:51 2022] PUT /identity/v3/projects/6cab1ea5ecc34b728f47c95d6fcc1d87/users/ba6483da3d7945cfafca7cae9082f4e7/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:51.857682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19434d91-b710-462b-94c0-829ef6680803 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:51.857962 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19434d91-b710-462b-94c0-829ef6680803 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:51.858255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19434d91-b710-462b-94c0-829ef6680803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:51.888590 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-19434d91-b710-462b-94c0-829ef6680803 None None] MFA Rules not processed for user `ba6483da3d7945cfafca7cae9082f4e7`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:51.889800 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-19434d91-b710-462b-94c0-829ef6680803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:51.956658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1454/2862] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:51.961374 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:51.992508 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:52.026234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac669c63-50fa-43c0-bdf1-80b6069e6559 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:52.030841 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:52.031441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.031716 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.031985 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] PATH_INFO: `/v3/projects` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.032607 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:52.034666 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:52.123212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1409/2863] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:51 2022] POST /identity/v3/projects => generated 373 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:52.126682 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19434d91-b710-462b-94c0-829ef6680803 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:52.137488 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19434d91-b710-462b-94c0-829ef6680803 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:52.197473 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:52.198177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.198359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.198665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] PATH_INFO: `/v3/users` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.199303 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:52.201233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:52.332941 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1455/2864] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:52 2022] POST /identity/v3/users => generated 293 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:52.336873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:52.347375 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aeeb7cf-34d2-49af-8d4c-5cbd51d5f76b admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:52.352261 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:52.352877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.353158 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.353428 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.354093 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:52.355782 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:52.367166 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1410/2865] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:52.376618 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:52.386767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6403736-a1f0-401b-9b1f-379291a9ee0c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:52.392106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:52.392766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.393248 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.393480 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] PATH_INFO: `/v3/projects/a3371c2152a64ef8adc130a1853fb06f/users/d2ba5ed9039b4715a19aa6d6e979d9a8/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.410507 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3371c2152a64ef8adc130a1853fb06f, user_id=d2ba5ed9039b4715a19aa6d6e979d9a8, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:52.413198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:52.485324 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1456/2866] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:52 2022] PUT /identity/v3/projects/a3371c2152a64ef8adc130a1853fb06f/users/d2ba5ed9039b4715a19aa6d6e979d9a8/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:52.489992 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:52.500482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9b0b99-cd17-40b9-beb6-1009dff98143 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:52.559873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c9edc9-1b60-4a41-b9f8-d3733e675ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:52.560548 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9edc9-1b60-4a41-b9f8-d3733e675ce3 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.560826 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9edc9-1b60-4a41-b9f8-d3733e675ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.561150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9edc9-1b60-4a41-b9f8-d3733e675ce3 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.562168 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c9edc9-1b60-4a41-b9f8-d3733e675ce3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:52.564377 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c9edc9-1b60-4a41-b9f8-d3733e675ce3 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:52.578180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1411/2867] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:52 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:52.579574 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:52.590681 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d1309e-d49f-4d15-801f-574a493957fe admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:52.595456 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:52.596641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.596916 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.597182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] PATH_INFO: `/v3/projects/a3371c2152a64ef8adc130a1853fb06f/users/d2ba5ed9039b4715a19aa6d6e979d9a8/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.599111 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3371c2152a64ef8adc130a1853fb06f, user_id=d2ba5ed9039b4715a19aa6d6e979d9a8, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:52.601711 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:52.616669 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1457/2868] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:52 2022] PUT /identity/v3/projects/a3371c2152a64ef8adc130a1853fb06f/users/d2ba5ed9039b4715a19aa6d6e979d9a8/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:52.628270 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.628498 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.628772 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.660298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] MFA Rules not processed for user `d2ba5ed9039b4715a19aa6d6e979d9a8`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:52.661623 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:52.734212 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1412/2869] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:52 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:52.748671 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:52.750626 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:52.788563 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:52.815845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22262280-51b2-43c0-b1a6-3b24d00ad859 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:52.819706 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:52.820547 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:52.820822 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:52.821090 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:52.883806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:52.885716 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:52.896256 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1458/2870] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:52 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:53.014789 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:53.016489 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:53.042930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:53.078457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569f5d88-408d-41ee-a7a1-4ecd9f07031f None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:53.082014 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:53.082674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:53.082949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:53.083399 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:53.085215 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:53.151326 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:53.152731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:53.163736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1413/2871] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:53 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:55.400423 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:55.401512 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:55.490095 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85733bdd-e068-4177-9057-33cb42f0e46b service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:55.496163 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1945e7-d12c-4c58-a967-9fba1b161d64 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:55.496781 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1945e7-d12c-4c58-a967-9fba1b161d64 service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:55.497060 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1945e7-d12c-4c58-a967-9fba1b161d64 service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:55.497336 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1945e7-d12c-4c58-a967-9fba1b161d64 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:55.499106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-6a1945e7-d12c-4c58-a967-9fba1b161d64 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:55.512142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1945e7-d12c-4c58-a967-9fba1b161d64 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:55.513726 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1945e7-d12c-4c58-a967-9fba1b161d64 service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:55.524851 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1459/2872] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:56.141320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:56.220526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc5a611-7bfc-4183-9e63-d4d75232adeb service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:56.226359 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:56.226947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:56.227328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:56.227600 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] PATH_INFO: `/v3/users/ba6483da3d7945cfafca7cae9082f4e7` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:56.229659 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba6483da3d7945cfafca7cae9082f4e7)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:56.231764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:56.437829 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba6483da3d7945cfafca7cae9082f4e7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:56.450035 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba6483da3d7945cfafca7cae9082f4e7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:56.450832 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba6483da3d7945cfafca7cae9082f4e7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:56.452081 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba6483da3d7945cfafca7cae9082f4e7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:56.532304 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba6483da3d7945cfafca7cae9082f4e7'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:56.541850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:56.557362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1414/2873] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:56 2022] DELETE /identity/v3/users/ba6483da3d7945cfafca7cae9082f4e7 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:56.562260 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:56.562764 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:56.562888 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:56.581966 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:56.583210 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:56.644085 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1460/2874] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:56.658772 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:56.660305 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:56.720048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d62cc1-6670-4e52-87ea-62d088b1ed90 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:56.724705 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:56.725331 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:56.725610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:56.725882 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:56.735609 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:56.738150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:56.747764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1415/2875] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:01:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:01:57.181147 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:57.182873 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:57.217587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88882239-7b8b-4198-adb1-6934f6fe9511 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:57.224564 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:57.224949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:57.225229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:57.225519 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] PATH_INFO: `/v3/projects/6cab1ea5ecc34b728f47c95d6fcc1d87` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:57.226665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=6cab1ea5ecc34b728f47c95d6fcc1d87)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:57.228679 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:57.231415 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=100 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:57.374252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6cab1ea5ecc34b728f47c95d6fcc1d87', 'request_id': 'req-49b4e34d-46f1-4105-854b-7144715088ce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.381741 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6cab1ea5ecc34b728f47c95d6fcc1d87', 'request_id': 'req-49b4e34d-46f1-4105-854b-7144715088ce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.459153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6cab1ea5ecc34b728f47c95d6fcc1d87', 'request_id': 'req-49b4e34d-46f1-4105-854b-7144715088ce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.472748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] The token cache is being invalidate because project 6cab1ea5ecc34b728f47c95d6fcc1d87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:57.474086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.474086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.475342 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1461/2876] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:57 2022] DELETE /identity/v3/projects/6cab1ea5ecc34b728f47c95d6fcc1d87 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:57.479295 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:57.489199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8add74-d35a-494e-a254-11ed8c387655 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:57.547240 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:57.547873 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:57.548156 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:57.548431 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] PATH_INFO: `/v3/users/d2ba5ed9039b4715a19aa6d6e979d9a8` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:57.549574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2ba5ed9039b4715a19aa6d6e979d9a8)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:57.551526 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:57.708877 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2ba5ed9039b4715a19aa6d6e979d9a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.715806 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2ba5ed9039b4715a19aa6d6e979d9a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.716460 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2ba5ed9039b4715a19aa6d6e979d9a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.717061 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2ba5ed9039b4715a19aa6d6e979d9a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.781584 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-ec6079e1-dd92-4a95-a6d9-db3feb2470d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2ba5ed9039b4715a19aa6d6e979d9a8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:57.799602 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1416/2877] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:01:57 2022] DELETE /identity/v3/users/d2ba5ed9039b4715a19aa6d6e979d9a8 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:58.203030 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:58.215106 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:58.219991 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b4e34d-46f1-4105-854b-7144715088ce admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:58.280978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:58.281632 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:58.282120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:58.282120 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] PATH_INFO: `/v3/projects/a3371c2152a64ef8adc130a1853fb06f` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:58.285016 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3371c2152a64ef8adc130a1853fb06f)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:58.285806 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:58.420009 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3371c2152a64ef8adc130a1853fb06f', 'request_id': 'req-d973173d-8c6b-43c3-bf0b-4a528d856c11'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:58.426799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3371c2152a64ef8adc130a1853fb06f', 'request_id': 'req-d973173d-8c6b-43c3-bf0b-4a528d856c11'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:58.515694 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3371c2152a64ef8adc130a1853fb06f', 'request_id': 'req-d973173d-8c6b-43c3-bf0b-4a528d856c11'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:58.527253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] The token cache is being invalidate because project a3371c2152a64ef8adc130a1853fb06f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:01:58.527542 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:58.528196 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-d973173d-8c6b-43c3-bf0b-4a528d856c11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:01:58.529695 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1462/2878] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:01:58 2022] DELETE /identity/v3/projects/a3371c2152a64ef8adc130a1853fb06f => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:58.535134 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71eef074-016c-42d8-a9dd-97a0d31d4135 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:58.535610 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71eef074-016c-42d8-a9dd-97a0d31d4135 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:58.535710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71eef074-016c-42d8-a9dd-97a0d31d4135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:58.556814 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-71eef074-016c-42d8-a9dd-97a0d31d4135 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:58.558249 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-71eef074-016c-42d8-a9dd-97a0d31d4135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:58.567189 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:58.623798 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1417/2879] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:58.631463 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e58545-d3f2-4c49-9713-2ef888af6c08 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:58.631748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e58545-d3f2-4c49-9713-2ef888af6c08 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:58.632019 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e58545-d3f2-4c49-9713-2ef888af6c08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:58.657438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-d5e58545-d3f2-4c49-9713-2ef888af6c08 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:58.659235 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-d5e58545-d3f2-4c49-9713-2ef888af6c08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:58.667033 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1463/2880] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:58.671361 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71eef074-016c-42d8-a9dd-97a0d31d4135 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:58.681707 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71eef074-016c-42d8-a9dd-97a0d31d4135 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:58.686301 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:58.686930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:58.687208 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:58.687479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] PATH_INFO: `/v3/domains` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:58.688121 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=88204) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 09:01:58.688418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:58.690227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:58.702394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1418/2881] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Oct 4 09:01:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 09:01:58.707374 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e58545-d3f2-4c49-9713-2ef888af6c08 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:58.716808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e58545-d3f2-4c49-9713-2ef888af6c08 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:58.722252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:58.722886 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:58.723159 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:58.723438 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:58.724049 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:58.725906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:58.740699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:58.825354 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1464/2882] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:58 2022] POST /identity/v3/projects => generated 383 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:58.829632 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:58.840241 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794a74e9-db63-40b2-b63a-8946a3a6610f admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:58.902635 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:58.903178 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:58.903502 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:58.903755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:58.904559 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:58.906595 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:58.998953 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1419/2883] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:58 2022] POST /identity/v3/users => generated 292 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:59.002988 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.013384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d388a180-9ec9-4bb3-8468-0459281518e5 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.019831 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.025154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.025154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.025154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.025154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.025154 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.034734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1465/2884] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:59.038835 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.048693 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038159a8-219f-4422-8f78-1af9e6ea5bd2 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.053869 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da89e04b-8919-4873-ad1b-a087ec2bac93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.054496 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89e04b-8919-4873-ad1b-a087ec2bac93 admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.054770 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89e04b-8919-4873-ad1b-a087ec2bac93 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.055036 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89e04b-8919-4873-ad1b-a087ec2bac93 admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.055670 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da89e04b-8919-4873-ad1b-a087ec2bac93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.057234 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da89e04b-8919-4873-ad1b-a087ec2bac93 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.069367 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1420/2885] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:59.073825 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.084153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3688b31c-4034-4cbd-a1db-b969d5a9da1c admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.088786 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.089425 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.089718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.089969 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] PATH_INFO: `/v3/projects/9e2640e306a844b5856938fdaf9f7812/users/559bf4dbf77d480c8b8f5ebe06cf821e/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.104188 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e2640e306a844b5856938fdaf9f7812, user_id=559bf4dbf77d480c8b8f5ebe06cf821e, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.106879 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.177500 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1466/2886] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:59 2022] PUT /identity/v3/projects/9e2640e306a844b5856938fdaf9f7812/users/559bf4dbf77d480c8b8f5ebe06cf821e/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:01:59.190843 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef95c0-f6af-4616-a679-8ccdb5f5d3e2 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.191209 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef95c0-f6af-4616-a679-8ccdb5f5d3e2 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.191483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef95c0-f6af-4616-a679-8ccdb5f5d3e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.218482 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-80ef95c0-f6af-4616-a679-8ccdb5f5d3e2 None None] MFA Rules not processed for user `559bf4dbf77d480c8b8f5ebe06cf821e`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:59.219691 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-80ef95c0-f6af-4616-a679-8ccdb5f5d3e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:01:59.287517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1421/2887] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:01:59.292285 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.354648 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d7552-92c4-41e7-8925-d1a4216faf47 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.359358 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.360076 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.360359 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.360629 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.361265 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.363762 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.491980 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1467/2888] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:01:59 2022] POST /identity/v3/projects => generated 383 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:59.496193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ef95c0-f6af-4616-a679-8ccdb5f5d3e2 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.512298 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ef95c0-f6af-4616-a679-8ccdb5f5d3e2 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.519928 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:01:59.594124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.594857 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.595077 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.595346 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.597157 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.600294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.696287 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1422/2889] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:01:59 2022] POST /identity/v3/users => generated 292 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:01:59.702337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.712670 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b45802b-96d7-415d-8473-5a8246ea76cd admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.717376 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.721490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.721490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.721490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.721490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.721490 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.731872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1468/2890] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:01:59.735881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.746858 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4ee027-50ef-4237-ad85-afa243ac4c75 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.751335 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35eb98b-8152-4e39-b3e3-cea739069c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.751947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35eb98b-8152-4e39-b3e3-cea739069c6b admin admin] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.752264 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35eb98b-8152-4e39-b3e3-cea739069c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.752576 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35eb98b-8152-4e39-b3e3-cea739069c6b admin admin] PATH_INFO: `/v3/roles` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.753193 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35eb98b-8152-4e39-b3e3-cea739069c6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.754902 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35eb98b-8152-4e39-b3e3-cea739069c6b admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.765767 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1423/2891] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:01:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:01:59.769862 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:01:59.779757 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174616b1-f972-4d3b-9737-d068dd99f48d admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:01:59.785051 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:01:59.785698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] REQUEST_METHOD: `PUT` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.785979 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.786247 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] PATH_INFO: `/v3/projects/4949eff87f0f4cef9795451355720a75/users/3829be21b82843d99f120b703e2d7675/roles/40b933f8a46e4be9bd70366ec14885e0` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.800992 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=4949eff87f0f4cef9795451355720a75, user_id=3829be21b82843d99f120b703e2d7675, role_id=40b933f8a46e4be9bd70366ec14885e0)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:01:59.803655 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:01:59.898557 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1469/2892] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:01:59 2022] PUT /identity/v3/projects/4949eff87f0f4cef9795451355720a75/users/3829be21b82843d99f120b703e2d7675/roles/40b933f8a46e4be9bd70366ec14885e0 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:01:59.910015 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b2fad7-5b25-4ba3-9962-6f00610536f5 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:01:59.915744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b2fad7-5b25-4ba3-9962-6f00610536f5 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:01:59.915744 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b2fad7-5b25-4ba3-9962-6f00610536f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:01:59.947173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-20b2fad7-5b25-4ba3-9962-6f00610536f5 None None] MFA Rules not processed for user `3829be21b82843d99f120b703e2d7675`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:01:59.948442 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-20b2fad7-5b25-4ba3-9962-6f00610536f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:00.014667 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1424/2893] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:01:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:02:00.020275 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:00.083565 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc53fa9-8237-4abe-8b81-f8f0a9c9c8fe admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:00.089437 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:00.090035 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.090310 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.090611 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] PATH_INFO: `/v3/projects` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.091229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:00.093119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:00.107522 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:00.179015 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1470/2894] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Oct 4 09:02:00 2022] POST /identity/v3/projects => generated 383 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:02:00.183254 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b2fad7-5b25-4ba3-9962-6f00610536f5 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:00.194567 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b2fad7-5b25-4ba3-9962-6f00610536f5 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:00.259545 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:00.260073 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.260349 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.260618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] PATH_INFO: `/v3/users` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.261267 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:00.263244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:00.372848 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1425/2895] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Oct 4 09:02:00 2022] POST /identity/v3/users => generated 298 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 09:02:00.377593 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:00.389387 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38afa91-f653-4bd4-b680-c205726d8c9f admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:00.395566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:00.396250 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.396491 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.396845 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.397489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:00.399333 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:00.410296 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1471/2896] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 09:02:00.414704 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:00.424720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d97cb-e63e-4312-a93f-ab4161dc0cd7 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:00.429214 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:00.429852 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.430126 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.430394 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] PATH_INFO: `/v3/projects/f1d5178845b148b1bd4f65cb9e8b9d88/users/4cd8772a231b4237802302c8d6ce9f90/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.432561 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:00.445702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1d5178845b148b1bd4f65cb9e8b9d88, user_id=4cd8772a231b4237802302c8d6ce9f90, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:00.448389 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:00.531038 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1426/2897] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:02:00 2022] PUT /identity/v3/projects/f1d5178845b148b1bd4f65cb9e8b9d88/users/4cd8772a231b4237802302c8d6ce9f90/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:02:00.538155 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:00.552964 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f8636c-2d1d-4338-babc-f83282627985 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:00.612278 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c44d21-4683-450a-9e84-1a63f85d0796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:00.612883 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c44d21-4683-450a-9e84-1a63f85d0796 admin admin] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.613176 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c44d21-4683-450a-9e84-1a63f85d0796 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.613433 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c44d21-4683-450a-9e84-1a63f85d0796 admin admin] PATH_INFO: `/v3/roles` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.614261 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c44d21-4683-450a-9e84-1a63f85d0796 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:00.615891 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c44d21-4683-450a-9e84-1a63f85d0796 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:00.625740 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1472/2898] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Oct 4 09:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 09:02:00.630818 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:00.641991 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7168ae7-a881-4290-8307-3be24dd57025 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:00.647364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:00.648013 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] REQUEST_METHOD: `PUT` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.648342 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.648611 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] PATH_INFO: `/v3/projects/f1d5178845b148b1bd4f65cb9e8b9d88/users/4cd8772a231b4237802302c8d6ce9f90/roles/1922dc727b87489999cc8916eacc229e` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.650520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1d5178845b148b1bd4f65cb9e8b9d88, user_id=4cd8772a231b4237802302c8d6ce9f90, role_id=1922dc727b87489999cc8916eacc229e)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:00.653164 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:00.663350 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1427/2899] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Oct 4 09:02:00 2022] PUT /identity/v3/projects/f1d5178845b148b1bd4f65cb9e8b9d88/users/4cd8772a231b4237802302c8d6ce9f90/roles/1922dc727b87489999cc8916eacc229e => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:02:00.673521 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d98af-77fb-4015-8c93-b1235f620855 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.673803 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d98af-77fb-4015-8c93-b1235f620855 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.674243 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d98af-77fb-4015-8c93-b1235f620855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.702093 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-bf0d98af-77fb-4015-8c93-b1235f620855 None None] MFA Rules not processed for user `4cd8772a231b4237802302c8d6ce9f90`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:02:00.703346 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-bf0d98af-77fb-4015-8c93-b1235f620855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:00.773212 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1473/2900] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:02:00 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:02:00.784687 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:00.786537 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:00.844755 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933626fe-269d-44c5-a900-04c35bb24f76 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:00.849150 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:00.849778 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:00.850058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:00.850334 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:00.915881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:00.917384 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:00.927000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1428/2901] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:02:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:02:02.828772 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d98af-77fb-4015-8c93-b1235f620855 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:02.838732 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d98af-77fb-4015-8c93-b1235f620855 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:02.842287 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:02.842948 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:02.843226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:02.843499 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:02.899150 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:02.900502 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:02.909728 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1474/2902] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:02:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:02:04.433710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:04.442313 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e759b41-cf9f-486c-bea6-19dccb798fd1 service cinder] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:04.445893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7709e5b-ea24-492e-9131-dbbbfb8b01bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f49a19f9484d4aa99567e66b4fc396ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:04.446514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7709e5b-ea24-492e-9131-dbbbfb8b01bb service cinder] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:04.446793 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7709e5b-ea24-492e-9131-dbbbfb8b01bb service cinder] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:04.447063 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7709e5b-ea24-492e-9131-dbbbfb8b01bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:04.455838 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7709e5b-ea24-492e-9131-dbbbfb8b01bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:04.457242 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7709e5b-ea24-492e-9131-dbbbfb8b01bb service cinder] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:04.466544 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1429/2903] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:02:04 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:02:09.122970 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:09.127166 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:09.184937 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddda801b-00fd-40c2-afd5-a1f624aab4d8 service cinder] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:09.191402 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:09.192182 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:09.192479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:09.192748 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] PATH_INFO: `/v3/users/559bf4dbf77d480c8b8f5ebe06cf821e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:09.193872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=559bf4dbf77d480c8b8f5ebe06cf821e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:09.195838 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:09.414489 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '559bf4dbf77d480c8b8f5ebe06cf821e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:09.421702 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '559bf4dbf77d480c8b8f5ebe06cf821e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:09.422364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '559bf4dbf77d480c8b8f5ebe06cf821e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:09.422999 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '559bf4dbf77d480c8b8f5ebe06cf821e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:09.494785 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '559bf4dbf77d480c8b8f5ebe06cf821e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:09.512932 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1475/2904] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:02:09 2022] DELETE /identity/v3/users/559bf4dbf77d480c8b8f5ebe06cf821e => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:02:09.517863 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:09.518356 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:09.518484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:09.541244 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:02:09.542952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:09.609086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1430/2905] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:02:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:02:09.618788 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:09.620253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:09.681086 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df60d57-d680-4b82-95f3-d27eeb864b0e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:09.684687 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:09.685337 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:09.685613 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:09.685882 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:09.695555 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:09.696976 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:09.706146 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1476/2906] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:02:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:02:10.150881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:10.150881 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:10.166939 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc88576a-38a6-4f1b-b29b-1733b46c6459 None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:10.176627 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:10.179993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:10.179993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:10.179993 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] PATH_INFO: `/v3/projects/9e2640e306a844b5856938fdaf9f7812` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:10.181425 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e2640e306a844b5856938fdaf9f7812)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:10.184713 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:10.194057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:10.308450 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e2640e306a844b5856938fdaf9f7812', 'request_id': 'req-24067e0c-81e6-435e-9d78-1d5481297cff'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.369651 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e2640e306a844b5856938fdaf9f7812', 'request_id': 'req-24067e0c-81e6-435e-9d78-1d5481297cff'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.376979 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e2640e306a844b5856938fdaf9f7812', 'request_id': 'req-24067e0c-81e6-435e-9d78-1d5481297cff'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.388199 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] The token cache is being invalidate because project 9e2640e306a844b5856938fdaf9f7812 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:02:10.388486 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.389137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.390960 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1431/2907] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:02:10 2022] DELETE /identity/v3/projects/9e2640e306a844b5856938fdaf9f7812 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:02:10.395659 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:10.405205 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c8ec25-88b3-4b9e-805d-34e517212391 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:10.425556 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:10.462523 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:10.463081 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:10.463362 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:10.463641 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] PATH_INFO: `/v3/users/3829be21b82843d99f120b703e2d7675` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:10.464766 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3829be21b82843d99f120b703e2d7675)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:10.466719 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:10.664198 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3829be21b82843d99f120b703e2d7675'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.671424 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3829be21b82843d99f120b703e2d7675'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.672091 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3829be21b82843d99f120b703e2d7675'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.672693 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3829be21b82843d99f120b703e2d7675'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.780686 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3829be21b82843d99f120b703e2d7675'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:10.800007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1477/2908] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:02:10 2022] DELETE /identity/v3/users/3829be21b82843d99f120b703e2d7675 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:02:11.138880 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:11.140463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:11.154117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24067e0c-81e6-435e-9d78-1d5481297cff admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:11.210104 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:11.210766 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:11.211047 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:11.211320 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] PATH_INFO: `/v3/users/7aeb97b19f12450fb3634eea8d5f1714` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:11.212500 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] RBAC: Authorizing `identity:delete_user(user_id=7aeb97b19f12450fb3634eea8d5f1714)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:11.214433 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:11.251007 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:11.252477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:11.266142 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599f85a2-e34f-4045-8e7d-51fa060e805e admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:11.271549 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:11.272195 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:11.272479 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:11.272747 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] PATH_INFO: `/v3/projects/4949eff87f0f4cef9795451355720a75` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:11.273875 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4949eff87f0f4cef9795451355720a75)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:11.275828 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:11.423598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4949eff87f0f4cef9795451355720a75', 'request_id': 'req-849e058f-3440-4758-ad67-57e2d7d9aa33'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.424781 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7aeb97b19f12450fb3634eea8d5f1714'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.431832 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4949eff87f0f4cef9795451355720a75', 'request_id': 'req-849e058f-3440-4758-ad67-57e2d7d9aa33'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.432718 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7aeb97b19f12450fb3634eea8d5f1714'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.433574 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7aeb97b19f12450fb3634eea8d5f1714'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.434722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7aeb97b19f12450fb3634eea8d5f1714'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.509177 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4949eff87f0f4cef9795451355720a75', 'request_id': 'req-849e058f-3440-4758-ad67-57e2d7d9aa33'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.520162 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] The token cache is being invalidate because project 4949eff87f0f4cef9795451355720a75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:02:11.520562 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.521381 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.522904 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1478/2909] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:02:11 2022] DELETE /identity/v3/projects/4949eff87f0f4cef9795451355720a75 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:02:11.526779 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:11.528136 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:11.566353 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:11.572352 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-da0e5f4f-0ab7-4604-a300-dc979503906d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7aeb97b19f12450fb3634eea8d5f1714'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.591477 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1432/2910] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:02:11 2022] DELETE /identity/v3/users/7aeb97b19f12450fb3634eea8d5f1714 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:02:11.595850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] REQUEST_METHOD: `POST` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:11.596139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:11.596413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:11.601122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849e058f-3440-4758-ad67-57e2d7d9aa33 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:11.617131 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.auth.core [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] MFA Rules not processed for user `e79e89b7261f4751ae9cd79132ff7ae2`. Rule list: `[]` (Enabled: `True`). {{(pid=88204) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:02:11.618353 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:11.626520 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:11.658884 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:11.659467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:11.659746 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:11.660011 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] PATH_INFO: `/v3/users/4cd8772a231b4237802302c8d6ce9f90` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:11.661658 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cd8772a231b4237802302c8d6ce9f90)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:11.663668 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:11.677996 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1433/2911] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:02:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:02:11.687897 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:11.689306 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:11.746668 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9796c0-3a1e-4551-9d0a-9c01e1adf59b None None] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:11.750289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:11.750918 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:11.751194 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:11.751470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:11.760514 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:11.761997 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:11.771977 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1434/2912] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:02:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 09:02:11.861498 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cd8772a231b4237802302c8d6ce9f90'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.869638 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cd8772a231b4237802302c8d6ce9f90'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.870384 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cd8772a231b4237802302c8d6ce9f90'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.871048 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cd8772a231b4237802302c8d6ce9f90'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.955900 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cd8772a231b4237802302c8d6ce9f90'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:11.977119 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1479/2913] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:02:11 2022] DELETE /identity/v3/users/4cd8772a231b4237802302c8d6ce9f90 => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:02:12.173692 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:12.175205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:12.193552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a9a62d-9fc9-408f-8828-63299981769a service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:12.259834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:12.260586 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:12.260865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:12.261173 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] PATH_INFO: `/v3/projects/3bb12f391098493b9bc69d7e7e5864d4` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:12.262314 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3bb12f391098493b9bc69d7e7e5864d4)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:12.264280 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:12.406492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3bb12f391098493b9bc69d7e7e5864d4', 'request_id': 'req-929310bb-58bb-43ad-884f-f39a29a8dff8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.422413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:02:12.423836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:02:12.436819 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eafa0c-cf43-4436-935d-4a3bc51f41e9 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:02:12.485368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:12.492174 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3bb12f391098493b9bc69d7e7e5864d4', 'request_id': 'req-929310bb-58bb-43ad-884f-f39a29a8dff8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.494527 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:12.495096 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:02:12.495738 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:02:12.496018 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:02:12.496289 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] PATH_INFO: `/v3/projects/f1d5178845b148b1bd4f65cb9e8b9d88` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:02:12.497447 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1d5178845b148b1bd4f65cb9e8b9d88)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:02:12.499227 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3bb12f391098493b9bc69d7e7e5864d4', 'request_id': 'req-929310bb-58bb-43ad-884f-f39a29a8dff8'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.499379 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:02:12.511533 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] The token cache is being invalidate because project 3bb12f391098493b9bc69d7e7e5864d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:02:12.511817 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.512470 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.514135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1435/2914] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:02:12 2022] DELETE /identity/v3/projects/3bb12f391098493b9bc69d7e7e5864d4 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:02:12.643262 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1d5178845b148b1bd4f65cb9e8b9d88', 'request_id': 'req-735a68dc-2453-4999-a27b-7b34903be828'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.649850 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1d5178845b148b1bd4f65cb9e8b9d88', 'request_id': 'req-735a68dc-2453-4999-a27b-7b34903be828'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.704343 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1d5178845b148b1bd4f65cb9e8b9d88', 'request_id': 'req-735a68dc-2453-4999-a27b-7b34903be828'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.719481 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] The token cache is being invalidate because project f1d5178845b148b1bd4f65cb9e8b9d88 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88203) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:02:12.719856 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.720477 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:02:12.722229 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1480/2915] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:02:12 2022] DELETE /identity/v3/projects/f1d5178845b148b1bd4f65cb9e8b9d88 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:02:23.066323 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=27,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:02:23.094626 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=4,keystone:DELETE=2 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:03:04.395972 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:04.397382 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:04.454451 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929310bb-58bb-43ad-884f-f39a29a8dff8 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:04.502069 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:04.502710 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:04.502988 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:04.503257 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:04.505000 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:04.563899 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:04.565355 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:04.574492 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1436/2916] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:05.489688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:05.491127 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:05.578482 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735a68dc-2453-4999-a27b-7b34903be828 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:05.583951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:05.584503 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:05.584794 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:05.585044 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:05.586951 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:05.599528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:05.601107 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:05.612949 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1481/2917] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:05.681726 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:05.684674 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:05.698364 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2a6b68-7259-40c5-9262-75d8a2c1eb1f service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:05.704884 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:05.705973 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:05.706483 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:05.706887 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:05.708842 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:05.724339 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:05.725783 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:05.737086 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1437/2918] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:05.949368 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:05.951020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:05.962700 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c84ba6-7b7f-4aaf-a986-3b73efe968af service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:05.966253 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:05.966881 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:05.967591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:05.967591 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:05.969268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:05.983453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:05.984945 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:05.996805 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1482/2919] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:07.183722 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:07.192413 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edd114d-da54-4ae9-8aca-fa972ff5a371 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:07.195946 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:07.196479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:07.196752 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:07.197017 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:07.198737 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:07.212018 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:07.213418 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:07.224679 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1438/2920] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:11.056802 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:11.066203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a15e8a-df4c-41b3-8281-2fbac8f2d375 service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:11.069874 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:11.073622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:11.073622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:11.073622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:11.073622 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:11.134967 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:11.136393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:11.146705 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1483/2921] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:17.219865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=80 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:03:18.128441 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:18.129850 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:18.207765 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07cb043-83b8-4f51-9760-5b11e4786f7d service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:18.212720 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:18.213328 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:18.213603 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:18.213875 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:18.216283 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:18.227702 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:18.229259 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:18.239989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1439/2922] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:18.271462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:18.272906 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:18.285833 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207cd611-7a54-4ede-a86f-ceccac08f4ac service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:18.290799 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:18.292321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:18.292321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:18.292321 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] PATH_INFO: `/v3/users/faa4a1f5f6954d92830b7df74952161e` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:18.293069 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] RBAC: Authorizing `identity:delete_user(user_id=faa4a1f5f6954d92830b7df74952161e)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:18.295814 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:18.451467 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'faa4a1f5f6954d92830b7df74952161e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:18.459279 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'faa4a1f5f6954d92830b7df74952161e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:18.459985 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'faa4a1f5f6954d92830b7df74952161e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:18.460583 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'faa4a1f5f6954d92830b7df74952161e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:18.464776 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:03:18.522808 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'faa4a1f5f6954d92830b7df74952161e'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:18.542420 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1484/2923] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:03:18 2022] DELETE /identity/v3/users/faa4a1f5f6954d92830b7df74952161e => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 09:03:18.770552 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:18.784484 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4aded1-a986-4c34-af3f-3b291fcc0195 service placement] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:18.841952 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:18.842618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:18.842895 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:18.843165 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] PATH_INFO: `/v3/projects/655f83bbd3d847f3b060af2ff5250585` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:18.844258 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=655f83bbd3d847f3b060af2ff5250585)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:18.846143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:18.963457 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '655f83bbd3d847f3b060af2ff5250585', 'request_id': 'req-95e9e058-e39f-4ab2-97da-3b4f21a674f1'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.048137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '655f83bbd3d847f3b060af2ff5250585', 'request_id': 'req-95e9e058-e39f-4ab2-97da-3b4f21a674f1'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.055119 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '655f83bbd3d847f3b060af2ff5250585', 'request_id': 'req-95e9e058-e39f-4ab2-97da-3b4f21a674f1'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.066553 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] The token cache is being invalidate because project 655f83bbd3d847f3b060af2ff5250585 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:03:19.066840 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.067479 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.068957 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1440/2924] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:03:18 2022] DELETE /identity/v3/projects/655f83bbd3d847f3b060af2ff5250585 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:03:19.073117 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:19.074610 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:19.088518 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119dda60-48eb-4ecf-8cea-bcff6fc79b5b admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:19.144627 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:19.145341 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:19.145620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:19.145892 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] PATH_INFO: `/v3/users/b641fda917cf44949c6d93f7e1946cce` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:19.147042 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b641fda917cf44949c6d93f7e1946cce)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:19.148957 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:19.307934 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b641fda917cf44949c6d93f7e1946cce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.315462 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b641fda917cf44949c6d93f7e1946cce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.316122 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b641fda917cf44949c6d93f7e1946cce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.316745 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b641fda917cf44949c6d93f7e1946cce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.373566 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-3dbe8736-71b4-4a25-bd00-19f583b1b273 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b641fda917cf44949c6d93f7e1946cce'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.391226 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1485/2925] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:03:19 2022] DELETE /identity/v3/users/b641fda917cf44949c6d93f7e1946cce => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:03:19.722620 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:19.724058 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:19.738578 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e9e058-e39f-4ab2-97da-3b4f21a674f1 admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:19.763113 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:03:19.818511 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:19.819188 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:19.819497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:19.819764 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] PATH_INFO: `/v3/projects/9241113fa00f46e5859c039629bc1f3d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:19.820911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9241113fa00f46e5859c039629bc1f3d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:19.822893 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:19.926143 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9241113fa00f46e5859c039629bc1f3d', 'request_id': 'req-b97d6958-e5d5-4311-9104-2b3978500b3e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.984944 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9241113fa00f46e5859c039629bc1f3d', 'request_id': 'req-b97d6958-e5d5-4311-9104-2b3978500b3e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:19.991624 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9241113fa00f46e5859c039629bc1f3d', 'request_id': 'req-b97d6958-e5d5-4311-9104-2b3978500b3e'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:20.003001 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] The token cache is being invalidate because project 9241113fa00f46e5859c039629bc1f3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:03:20.003282 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:20.003909 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:03:20.005385 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1441/2926] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:03:19 2022] DELETE /identity/v3/projects/9241113fa00f46e5859c039629bc1f3d => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:03:29.387487 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=61,keystone:DELETE=12,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:03:29.999947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:03:57.578043 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:57.578698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:57.578698 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:57.598699 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] MFA Rules not processed for user `471299b556bd4c22b51defc14b49d8eb`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:03:57.599909 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:57.656528 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1486/2927] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:03:57 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:03:57.966517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:57.967929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:58.024989 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97d6958-e5d5-4311-9104-2b3978500b3e admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:58.028547 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:58.029153 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:58.029447 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:58.029729 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:58.031517 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:58.041129 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:58.042571 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:58.050657 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1442/2928] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:03:58.819178 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:03:58.820680 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:58.892733 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0193d8f7-e313-49e9-b3bb-f6112a14c7a7 None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:03:58.897620 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e560d0c-c795-49d2-9562-ede6352afd46 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:03:58.898224 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e560d0c-c795-49d2-9562-ede6352afd46 service placement] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:03:58.898527 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e560d0c-c795-49d2-9562-ede6352afd46 service placement] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:03:58.898809 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e560d0c-c795-49d2-9562-ede6352afd46 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:03:58.900504 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-4e560d0c-c795-49d2-9562-ede6352afd46 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:03:58.912133 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e560d0c-c795-49d2-9562-ede6352afd46 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:03:58.913685 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e560d0c-c795-49d2-9562-ede6352afd46 service placement] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:03:58.924026 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1487/2929] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:03:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:04:08.047089 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=26 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:04:08.919153 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=57 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:04:13.050949 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:04:13.052449 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:04:13.127682 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e571f9d-bb57-48f5-abd9-7e6ee856d1a7 service neutron] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:04:13.133971 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:04:13.134618 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:04:13.134900 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:04:13.135172 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:04:13.137497 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:04:13.151135 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:04:13.152773 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:04:13.164546 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1443/2930] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:04:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:04:23.160472 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=34 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:05:52.941836 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfc642-070b-4fad-9182-901257b5cdca None None] REQUEST_METHOD: `POST` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:05:52.942393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfc642-070b-4fad-9182-901257b5cdca None None] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:05:52.942393 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfc642-070b-4fad-9182-901257b5cdca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:05:52.961996 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.auth.core [None req-a1dfc642-070b-4fad-9182-901257b5cdca None None] MFA Rules not processed for user `471299b556bd4c22b51defc14b49d8eb`. Rule list: `[]` (Enabled: `True`). {{(pid=88203) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 09:05:52.963236 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-a1dfc642-070b-4fad-9182-901257b5cdca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:05:52.970233 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1488/2931] 10.222.0.27 () {58 vars in 958 bytes} [Tue Oct 4 09:05:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 09:05:54.944856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:05:54.947006 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:05:54.960819 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720b1f8-51d5-43af-bfb3-6d53b47542a6 service nova] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:05:54.966469 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:05:54.967084 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:05:54.967362 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:05:54.967631 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:05:54.969275 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:05:54.980499 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:05:54.982067 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:05:54.992463 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1444/2932] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:05:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:06:02.956414 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=3 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:06:04.987768 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=6 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:06:06.829207 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfc642-070b-4fad-9182-901257b5cdca None None] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:06:06.842364 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfc642-070b-4fad-9182-901257b5cdca None None] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:06:06.847983 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:06:06.848587 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:06:06.848867 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:06:06.849139 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:06:06.850954 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:06:06.862200 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:06:06.863807 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:06:06.873187 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1489/2933] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:06:06 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:06:16.869982 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=6 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:07:42.735139 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:07:42.736769 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:07:42.753205 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faee3f3-51fa-4887-b6f2-743ece02ba0f service placement] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:07:42.759125 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '40e12aa547b84ac38fd222a9f5f62ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:07:42.768757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:07:42.768757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:07:42.768757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:07:42.768757 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:07:42.778144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:07:42.779911 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:07:42.792028 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1445/2934] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:07:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:07:52.786294 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=6 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:07:57.924588 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:07:57.940000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ffef2a-22e2-4176-a955-8a5795252f4b service nova] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:07:57.940000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df391d2bf7884538823f04db5564d02c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:07:57.940000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:07:57.940000 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:07:57.940624 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:07:57.940872 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:07:58.004020 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:07:58.005455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:07:58.014978 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1490/2935] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:07:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:08:01.536231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:08:01.548180 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80da820c-bcb7-4066-ab32-a38e76885705 service nova] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:08:01.554056 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:08:01.554699 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] REQUEST_METHOD: `GET` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:08:01.554976 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:08:01.555735 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:08:01.557144 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:08:01.569487 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:08:01.570990 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:08:01.581124 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1446/2936] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:08:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:08:05.580169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:08:05.580169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:08:05.592268 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330b3c21-e59c-48c3-8b62-d36155e74ded service neutron] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:08:05.596767 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:08:05.597455 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:08:05.597734 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:08:05.598003 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] PATH_INFO: `/v3/users/471299b556bd4c22b51defc14b49d8eb` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:08:05.599165 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=471299b556bd4c22b51defc14b49d8eb)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:08:05.601075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:08:05.812657 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '471299b556bd4c22b51defc14b49d8eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:05.819791 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '471299b556bd4c22b51defc14b49d8eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:05.820453 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '471299b556bd4c22b51defc14b49d8eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:05.821075 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '471299b556bd4c22b51defc14b49d8eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:05.901144 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '471299b556bd4c22b51defc14b49d8eb'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:05.919612 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1491/2937] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:08:05 2022] DELETE /identity/v3/users/471299b556bd4c22b51defc14b49d8eb => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:08:06.154109 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:08:06.166851 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117cf32-76e6-45e2-9cf2-1d8bde9dc6bb service placement] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:08:06.229930 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:08:06.230731 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:08:06.231012 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:08:06.231285 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] PATH_INFO: `/v3/projects/07b9220d86234adc83ef7fbe0d4d834d` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:08:06.232437 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=07b9220d86234adc83ef7fbe0d4d834d)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:08:06.234333 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:08:06.341137 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07b9220d86234adc83ef7fbe0d4d834d', 'request_id': 'req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.410920 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07b9220d86234adc83ef7fbe0d4d834d', 'request_id': 'req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.418929 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07b9220d86234adc83ef7fbe0d4d834d', 'request_id': 'req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.431633 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] The token cache is being invalidate because project 07b9220d86234adc83ef7fbe0d4d834d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:08:06.431914 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.432583 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.434057 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1447/2938] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:08:06 2022] DELETE /identity/v3/projects/07b9220d86234adc83ef7fbe0d4d834d => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:08:06.438217 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:08:06.439660 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:08:06.452721 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dbe86e-f657-4511-9e9d-1e8d0f35f8e6 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:08:06.508258 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:08:06.508921 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:08:06.509203 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:08:06.509476 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] PATH_INFO: `/v3/users/74bbf6ce6700410bafd4510241993345` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:08:06.510665 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74bbf6ce6700410bafd4510241993345)` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:08:06.512625 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:08:06.518761 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:08:06.703598 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74bbf6ce6700410bafd4510241993345'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.711036 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74bbf6ce6700410bafd4510241993345'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.711688 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74bbf6ce6700410bafd4510241993345'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.712293 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74bbf6ce6700410bafd4510241993345'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.803712 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.notifications [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74bbf6ce6700410bafd4510241993345'} {{(pid=88203) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:06.821185 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1492/2939] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Oct 4 09:08:06 2022] DELETE /identity/v3/users/74bbf6ce6700410bafd4510241993345 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:08:07.074865 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Authenticating user token {{(pid=88204) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:08:07.076289 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.fernet_utils [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88204) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:08:07.092117 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd0d23a-a5f8-47b4-9441-8d26d2d3da7c admin admin] Validating token access rules against request {{(pid=88204) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:08:07.148424 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e79e89b7261f4751ae9cd79132ff7ae2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '24e9902302e04e8e8d4caf3bfee360d5', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88204) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:08:07.149048 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:08:07.149324 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:08:07.149592 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] PATH_INFO: `/v3/projects/33c49c7ca4e44d45b2416ab67784f0eb` {{(pid=88204) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:08:07.150736 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33c49c7ca4e44d45b2416ab67784f0eb)` {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:08:07.152596 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] RBAC: Authorization granted {{(pid=88204) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:08:07.172917 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:08:07.265834 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33c49c7ca4e44d45b2416ab67784f0eb', 'request_id': 'req-74be0686-9c09-4dce-937b-71cb3790bbc4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:07.335223 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33c49c7ca4e44d45b2416ab67784f0eb', 'request_id': 'req-74be0686-9c09-4dce-937b-71cb3790bbc4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:07.341807 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33c49c7ca4e44d45b2416ab67784f0eb', 'request_id': 'req-74be0686-9c09-4dce-937b-71cb3790bbc4'} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:07.351947 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] The token cache is being invalidate because project 33c49c7ca4e44d45b2416ab67784f0eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=88204) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 09:08:07.352231 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:07.352856 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG keystone.notifications [None req-74be0686-9c09-4dce-937b-71cb3790bbc4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=88204) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 09:08:07.354316 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: [pid: 88204|app: 0|req: 1448/2940] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Oct 4 09:08:07 2022] DELETE /identity/v3/projects/33c49c7ca4e44d45b2416ab67784f0eb => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 09:08:16.817718 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:08:17.349045 dv-cinder-859447-2-iscsi devstack@keystone.service[88204]: DEBUG dbcounter [-] [88204] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=88204) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 09:11:16.040169 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Authenticating user token {{(pid=88203) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 09:11:16.041594 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:11:16.179933 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56bf24b-230c-4704-9ef7-7f8ddd0b15e2 admin admin] Validating token access rules against request {{(pid=88203) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 09:11:16.241815 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971ed03e-e512-4f98-95bf-695405a21c71 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '659e82e28da54e28a58d9d6e9c5e4215', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1b4ac0484ac7495f9bbc566d50e63fcf', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=88203) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 09:11:16.242830 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ed03e-e512-4f98-95bf-695405a21c71 service placement] REQUEST_METHOD: `GET` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 09:11:16.243112 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ed03e-e512-4f98-95bf-695405a21c71 service placement] SCRIPT_NAME: `/identity` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 09:11:16.243383 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ed03e-e512-4f98-95bf-695405a21c71 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=88203) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 09:11:16.245113 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.fernet_utils [None req-971ed03e-e512-4f98-95bf-695405a21c71 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=88203) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 09:11:16.256413 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971ed03e-e512-4f98-95bf-695405a21c71 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 09:11:16.260252 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971ed03e-e512-4f98-95bf-695405a21c71 service placement] RBAC: Authorization granted {{(pid=88203) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 09:11:16.269255 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: [pid: 88203|app: 0|req: 1493/2941] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Oct 4 09:11:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 09:11:26.264860 dv-cinder-859447-2-iscsi devstack@keystone.service[88203]: DEBUG dbcounter [-] [88203] Writing DB stats keystone:SELECT=89 {{(pid=88203) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}